Method, apparatus, and computer program product for machine learning model lifecycle management

ABSTRACT

Computing systems, computing apparatuses, computing methods, and computer program products are disclosed for machine learning model lifecycle management. An example computing method includes receiving a machine learning model selection, a machine learning model experiment creation input, a machine learning model experiment run type, and a machine learning model input data path. The example method further includes determining a machine learning model execution engine based on the machine learning model experiment creation input and the machine learning model experiment run type. The example method further includes retrieving input data based on the machine learning model input data path. The example method further includes executing a machine learning model experiment based on the machine learning model execution engine, machine learning model experiment creation input, and the input data. The example method further includes generating one or more machine learning model scores based on the machine learning model experiment.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisionalapplication Ser. No. 16/105,525, entitled “Method, Apparatus, AndComputer Program Product For Machine Learning Model LifecycleManagement,” filed Aug. 20, 2018, which claims priority to U.S.Provisional Application No. 62/547,420, entitled “Method, Apparatus, AndComputer Program Product For Machine Learning Model LifecycleManagement,” filed Aug. 18, 2017, the contents of each which areincorporated by reference herein in its entirety.

BACKGROUND

The inventors have discovered problems with existing mechanisms formachine learning model lifecycle management. Through applied effort,ingenuity, and innovation, the inventors has solved many of theseidentified problems by developing solutions embodied by the presentdisclosure and described in detail below.

BRIEF SUMMARY

Systems, apparatuses, methods, and computer program products aredisclosed herein for machine learning model lifecycle management. Themachine learning model lifecycle management system provided hereinsolves the above problems by managing the lifecycle of machine learningmodels to include the execution and publication of machine learningmodel experiments and machine learning model scores.

In one example embodiment, a computing system is provided for machinelearning model lifecycle management. The computing system may compriseapplication programming interface (API) circuitry configured to receivea machine learning model selection. The API circuitry may be furtherconfigured to receive a machine learning model experiment creationinput. The machine learning model experiment creation input may comprisea machine learning model experiment name, a machine learning modelexperiment description, a machine learning model execution engineselection, at least one machine learning model scoring file, and atleast one machine learning model scoring data file. The API circuitrymay be further configured to receive a machine learning model experimentrun type. The machine learning model experiment run type may be one ofbatch scoring, real-time scoring, or training. The API circuitry may befurther configured to receive a machine learning model input data path.The computing system may comprise machine learning model lifecyclemanagement circuitry configured to determine a machine learning modelexecution engine based on the machine learning model execution engineselection and the machine learning model experiment run type. Themachine learning model lifecycle management circuitry may be furtherconfigured to retrieve input data based on the machine learning modelinput data path. The machine learning model lifecycle managementcircuitry may be further configured to execute a machine learning modelexperiment based on the machine learning model execution engine, themachine learning model scoring file, the at least one machine learningmodel scoring data file, and the input data. The machine learning modellifecycle management circuitry may be further configured to generate oneor more machine learning model scores based on the machine learningmodel experiment. The one or more machine learning model scores may beassociated with the machine learning model experiment and the inputdata.

In another example embodiment, a computing apparatus is provided formachine learning model lifecycle management. The computing apparatus maycomprise application programming interface (API) circuitry configured toreceive a machine learning model selection. The API circuitry may befurther configured to receive a machine learning model experimentcreation input. The machine learning model experiment creation input maycomprise a machine learning model experiment name, a machine learningmodel experiment description, a machine learning model execution engineselection, at least one machine learning model scoring file, and atleast one machine learning model scoring data file. The API circuitrymay be further configured to receive a machine learning model experimentrun type. The machine learning model experiment run type may be one ofbatch scoring, real-time scoring, or training. The API circuitry may befurther configured to receive a machine learning model input data path.The computing apparatus may comprise machine learning model lifecyclemanagement circuitry configured to determine a machine learning modelexecution engine based on the machine learning model execution engineselection and the machine learning model experiment run type. Themachine learning model lifecycle management circuitry may be furtherconfigured to retrieve input data based on the machine learning modelinput data path. The machine learning model lifecycle managementcircuitry may be further configured to execute a machine learning modelexperiment based on the machine learning model execution engine, themachine learning model scoring file, the at least one machine learningmodel scoring data file, and the input data. The machine learning modellifecycle management circuitry may be further configured to generate oneor more machine learning model scores based on the machine learningmodel experiment. The one or more machine learning model scores may beassociated with the machine learning model experiment and the inputdata.

In another example embodiment, a computing method is provided formachine learning model lifecycle management. The computing method maycomprise receiving, by application programming interface (API)circuitry, a machine learning model selection. The computing method mayfurther comprise receiving, by the API circuitry, a machine learningmodel experiment creation input. The machine learning model experimentcreation input may comprise a machine learning model experiment name, amachine learning model experiment description, a machine learning modelexecution engine selection, at least one machine learning model scoringfile, and at least one machine learning model scoring data file. Thecomputing method may further comprise receiving, by the API circuitry, amachine learning model experiment run type. The machine learning modelexperiment run type may be one of batch scoring, real-time scoring, ortraining. The computing method may further comprise receiving, by theAPI circuitry, a machine learning model input data path. The computingmethod may further comprise determining, by machine learning modellifecycle management circuitry, a machine learning model executionengine based on the machine learning model execution engine selectionand the machine learning model experiment run type. The computing methodmay further comprise retrieving, by the machine learning model lifecyclemanagement circuitry, input data based on the machine learning modelinput data path. The computing method may further comprise executing, bythe machine learning model lifecycle management circuitry, a machinelearning model experiment based on the machine learning model executionengine, the machine learning model scoring file, the at least onemachine learning model scoring data file, and the input data. Thecomputing method may further comprise generating, by the machinelearning model lifecycle management circuitry, one or more machinelearning model scores based on the machine learning model experiment.The one or more machine learning model scores may be associated with themachine learning model experiment and the input data.

In another example embodiment, a computer program product is providedfor machine learning model lifecycle management. The computer programproduct may comprise at least one non-transitory computer-readablestorage medium storing computer-executable program code instructionsthat, when executed by a computing system, cause the computing system toreceive, by application programming interface (API) circuitry, a machinelearning model selection. The computer-executable program codeinstructions, when executed by the computing system, may further causethe computing system to receive, by the API circuitry, a machinelearning model experiment creation input. The machine learning modelexperiment creation input may comprise a machine learning modelexperiment name, a machine learning model experiment description, amachine learning model execution engine selection, at least one machinelearning model scoring file, and at least one machine learning modelscoring data file. The computer-executable program code instructions,when executed by the computing system, may further cause the computingsystem to receive, by the API circuitry, a machine learning modelexperiment run type. The machine learning model experiment run type maybe one of batch scoring, real-time scoring, or training. Thecomputer-executable program code instructions, when executed by thecomputing system, may further cause the computing system to receive, bythe API circuitry, a machine learning model input data path. Thecomputer-executable program code instructions, when executed by thecomputing system, may further cause the computing system to determine,by machine learning model lifecycle management circuitry, a machinelearning model execution engine based on the machine learning modelexecution engine selection and the machine learning model experiment runtype. The computer-executable program code instructions, when executedby the computing system, may further cause the computing system toretrieve, by the machine learning model lifecycle management circuitry,input data based on the machine learning model input data path. Thecomputer-executable program code instructions, when executed by thecomputing system, may further cause the computing system to execute, bythe machine learning model lifecycle management circuitry, a machinelearning model experiment based on the machine learning model executionengine, the machine learning model scoring file, the at least onemachine learning model scoring data file, and the input data. Thecomputer-executable program code instructions, when executed by thecomputing system, may further cause the computing system to generate, bythe machine learning model lifecycle management circuitry, one or moremachine learning model scores based on the machine learning modelexperiment. The one or more machine learning model scores may beassociated with the machine learning model experiment and the inputdata.

The foregoing brief summary is provided merely for purposes ofsummarizing some example embodiments illustrating some aspects of thepresent disclosure. Accordingly, it will be appreciated that theabove-described embodiments are merely examples and should not beconstrued to narrow the scope of the present disclosure in any way. Itwill be appreciated that the scope of the present disclosure encompassesmany potential embodiments in addition to those summarized herein, someof which will be described in further detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are not necessarily drawn to scale,illustrate embodiments and features of the present disclosure. Togetherwith the specification, including the brief summary above and thedetailed description below, the accompanying drawings serve to explainthe embodiments and features of the present disclosure. The componentsillustrated in the figures represent components that may or may not bepresent in various embodiments or features of the disclosure describedherein. Accordingly, some embodiments or features of the presentdisclosure may include fewer or more components than those shown in thedrawings while not departing from the scope of the disclosure.

FIG. 1A illustrates an example system diagram in accordance with someexample embodiments;

FIG. 1B illustrates an example logical design diagram of a system inaccordance with some example embodiments;

FIG. 1C illustrates an example design diagram of a system in accordancewith some example embodiments;

FIG. 1D illustrates an example architecture diagram of a system inaccordance with some example embodiments;

FIG. 2 illustrates an example schematic block diagram in accordance withsome example embodiments;

FIGS. 3A and 3B illustrate example machine learning model lifecyclemanagement system architecture data models in accordance with someexample embodiments;

FIG. 4 illustrates an example architecture diagram of an example featurecatalog system in accordance with some example embodiments;

FIG. 5 illustrates an example data model of an example feature catalogsystem in accordance with some example embodiments;

FIG. 6 illustrates an example machine learning model lifecyclemanagement process in accordance with some example embodiments;

FIG. 7 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 8 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 9 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 10 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 11 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 12 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 13 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 14 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 15 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 16 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 17 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 18 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 19 illustrates an example user interface display screen inaccordance with some example embodiments;

FIG. 20 illustrates an example user interface display screen inaccordance with some example embodiments; and

FIG. 21 illustrates an example user interface display screen inaccordance with some example embodiments.

DETAILED DESCRIPTION

Some embodiments of the present disclosure will now be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the disclosures are shown. Indeed,these disclosures may be embodied in many different forms and should notbe construed as limited to the embodiments set forth herein; rather,these embodiments are provided so that this disclosure will satisfyapplicable legal requirements. The term “or” is used herein in both thealternative and conjunctive sense, unless otherwise indicated. The terms“illustrative” and “example” are used to be examples with no indicationof quality level. Like numbers refer to like elements throughout.

Overview

As noted above, computing systems, computing apparatuses, computingmethods, and computer program products are described herein that providefor machine learning model lifecycle management. Machine learning is asubfield of computer science that gives computers the ability to learnwithout being explicitly programmed. Evolved from the study of patternrecognition and computational learning theory in artificialintelligence, machine learning explores the study and construction ofalgorithms that may learn from and make predictions on data. Suchalgorithms overcome following strictly static program instructions bymaking data-driven predictions or decisions through building a machinelearning model from sample inputs. Machine learning is employed in arange of computing tasks where designing and programming explicitalgorithms with good performance is difficult or infeasible.

Machine learning enables prediction-making through the use ofspecialized circuitry, computers, or both. Machine learning may beunsupervised for exploratory data analysis. Machine learning may also beunsupervised and be used to learn and establish baseline behavioralprofiles for various entities and then used to find meaningfulanomalies. Machine learning is used to devise complex models andalgorithms that lend themselves to prediction; also known as predictiveanalytics. These machine learning models allow researchers, datascientists, engineers, and analysts to produce reliable, repeatabledecisions and results, and uncover hidden insights through learning fromhistorical relationships and trends in the data.

Machine learning model lifecycle management involves organizing variousexperiments for producing a certain prediction. Feedback on theperformance of the various experiments must be obtainable at thetraining phase as well as during and/or from live scoring runs. Based onthe performance feedback, the best performing experiments may beselected for use in making programmatic decisions downstream from themachine learning model.

In a production environment, systems downstream from machine learningmodels make decisions based upon predictions (also referred to herein as“scores”) produced by the machine learning models. The downstreamsystems must be able to trust that the machine learning models have beentrained using accurate and reliable data, that the machine learningmodels have been executed using accurate and reliable data, that themachine learning models are accurate and/or have a high level ofhistorical accuracy, and that the machine learning models have beenrecently verified to be accurate. The system resources and timeallocated to providing such assurances to downstream systems are easilyexhausted and compromised as a result of these challenges. The abilityto enable data scientists to leverage a scalable architecture to executemachine learning tasks on very large datasets may reduce the computingresources and time necessary to release new machine learning modelsand/or improvements to existing machine learning models into production.

With the present disclosure, multiple machine learning model experimentsmay be run simultaneously on very large datasets. Scores or predictionsproduced by machine learning model experiments may be selectivelypublished for consumption by downstream systems.

Various embodiments of the disclosure generally relate to machinelearning model lifecycle management. For example, the systems,apparatuses, methods, and computer program products described herein areoperable to receive a machine learning model selection and to receive amachine learning model experiment creation input. The machine learningmodel creation input comprises a machine learning model experiment name,a machine learning model experiment description, a machine learningmodel execution engine selection, at least one machine learning modelscoring file, and at least one machine learning model scoring data file.The systems, apparatuses, methods, and computer program productsdescribed herein are further operable to receive a machine learningmodel experiment run type and a machine learning model input data path,where the machine learning model experiment run type is one of trainingor batch scoring. The systems, apparatuses, methods, and computerprogram products described herein are further operable to execute amachine learning model experiment using the machine learning modelscoring file, the at least one machine learning model scoring data file,and input data retrieved from the machine learning model input data pathas inputs to a machine learning model execution engine provided by themachine learning model execution engine selection. The systems,apparatuses, methods, and computer program products described herein arefurther operable to generate one or more machine learning model scores,where the one or more machine learning model scores are associated withthe machine learning model experiment and the input data.

In one example embodiment, a computing entity or apparatus is configuredto receive a machine learning model selection; receive a machinelearning model experiment creation input, the machine learning modelcreation input comprising a machine learning model experiment name, amachine learning model experiment description, a machine learning modelexecution engine selection, at least one machine learning model scoringfile, and at least one machine learning model scoring data file; receivea machine learning model experiment run type and a machine learningmodel input data path, wherein the machine learning model experiment runtype is one of training or batch scoring; execute a machine learningmodel experiment using the machine learning model scoring file, the atleast one machine learning model scoring data file, and input dataretrieved from the machine learning model input data path as inputs to amachine learning model execution engine provided by the machine learningmodel execution engine selection; and generate one or more machinelearning model scores, wherein the one or more machine learning modelscores are associated with the machine learning model experiment and theinput data.

Other embodiments include corresponding systems, methods, and computerprograms, configured to perform the operations of the apparatus, encodedon computer storage devices.

Other embodiments may each optionally include where the at least onemachine learning model scoring data file comprises a trained machinelearning model.

Other embodiments may each optionally include where the at least oneprocessor and at least one memory including computer program code, theat least one memory and the computer program code configured to, withthe at least one processor, further cause the apparatus to: receive amachine learning model experiment publishing selection; and enableaccess to the machine learning model experiment and the one or moremachine learning model scores associated with the machine learning modelexperiment and the input data.

Other embodiments may each optionally include where executing themachine learning model experiment comprises: retrieving the input datastored in a memory location identified by the machine learning modelinput data path; loading the at least one machine learning model scoringdata file into a machine learning model execution environment, themachine learning model execution environment based on the machinelearning model execution engine selection; executing the machinelearning model scoring file to create a machine learning model dataframe comprising one or more machine learning model scores associatedwith the machine learning model experiment and the input data.

Other embodiments may each optionally include where the input datacomprises a feature vector, the feature vector comprising a most recentobservation for each of a plurality of features, each feature identifiedby a feature identifier and associated with an observation timestamp.

Other embodiments may each optionally include where each score statisticof the one or more machine learning model scores comprises a featureidentifier, a score timestamp, a score, and an experiment identifier.

Other embodiments may each optionally include where the trained machinelearning model is trained using a machine learning model trainingdataset, the machine learning model training dataset comprising aplurality of observations of a target variable and a feature vectorcomprising a plurality of associated observations of non-targetvariables observed at a time relative to the target variable.

Other embodiments may each optionally include where enabling access tothe machine learning model experiment and the one or more machinelearning model scores associated with the machine learning modelexperiment and the input data comprises publishing the machine learningmodel experiment, the one or more machine learning model scores, and oneor more machine learning model statistics in a feature catalog. Otherembodiments may each optionally include where the feature catalogcomprises a plurality of feature definitions.

There are many advantages of these and other embodiments describedherein, such as: providing for integration of offline model prototypesinto online production environments in a less time-consuming manner thatis also less prone to errors; providing a mechanism that versions thecontract between data scientists and engineers, enabling fast iterationwhile facilitating the maintenance of a stable production environment;supporting, depending on the use case, an offline batch execution or areal-time or streaming paradigm; providing a machine learning systemthat is easily scalable and able to handle loads that vary drasticallyin certain scenarios; providing a machine learning system that isextensible and able to onboard new machine learning libraries with ease;providing monitoring and archiving functionalities to enable keyactivities such as AB testing of models and offline analytics; andexpediting the deployment, execution, and monitoring of machine learningmodels.

Definitions

As used herein, the terms “data,” “content,” “information,” and similarterms may be used interchangeably to refer to data capable of beingtransmitted, received, and/or stored in accordance with embodiments ofthe present disclosure. Thus, use of any such terms should not be takento limit the spirit and scope of embodiments of the present disclosure.Further, where a computing device is described herein to receive datafrom another computing device, it will be appreciated that the data maybe received directly from another computing device or may be receivedindirectly via one or more intermediary computing devices, such as, forexample, one or more servers, relays, routers, network access points,base stations, hosts, and/or the like, sometimes referred to herein as a“network.” Similarly, where a computing device is described herein tosend data to another computing device, it will be appreciated that thedata may be sent directly to another computing device or may be sentindirectly via one or more intermediary computing devices, such as, forexample, one or more servers, cloud utilities, relays, routers, networkaccess points, base stations, hosts, and/or the like.

The term “comprising” means including but not limited to, and should beinterpreted in the manner it is typically used in the patent context.Use of broader terms such as comprises, includes, and having should beunderstood to provide support for narrower terms such as consisting of,consisting essentially of, and comprised substantially of.

The phrases “in some embodiments,” “in other embodiments,” “in someinstances,” and the like generally mean that the particular feature,structure, or characteristic following the phrase may be included in atleast one embodiment of the present disclosure, and may be included inmore than one embodiment of the present disclosure (e.g., such phrasesmay not necessarily refer to the same embodiment).

The word “example” is used herein to mean “serving as an example,instance, or illustration.” Any implementation described herein as“example” is not necessarily to be construed as preferred oradvantageous over other implementations.

If the specification states a component or feature “may,” “can,”“could,” “should,” “typically,” “optionally,” “generally,”“additionally,” “alternatively,” “for example,” “often,” or “might” (orother such language) be included or have a characteristic, thatparticular component or feature is not required to be included or tohave the characteristic. Such component or feature may be optionallyincluded in some embodiments, or it may be excluded.

The terms “processor” and “processing circuitry” are used herein torefer to any programmable microprocessor, microcomputer or multipleprocessor chip or chips that can be configured by software instructions(e.g., applications) to perform a variety of functions, including thefunctions of the various embodiments described above. In some devices,multiple processors may be provided, such as one processor dedicated towireless communication functions and one processor dedicated to runningother applications. Software applications may be stored in the internalmemory before they are accessed and loaded into the processors. Theprocessors may include internal memory sufficient to store theapplication software instructions. In many devices the internal memorymay be a volatile or nonvolatile memory, such as flash memory, or amixture of both. The memory may also be located internal to anothercomputing resource (e.g., enabling computer-readable instructions to bedownloaded over the Internet or another wired or wireless connection).

For the purposes of this description, a general reference to “memory”refers to memory accessible by the processors including internal memoryor removable memory plugged into the device, remote memory (e.g., cloudstorage), memory within the processors themselves, or a combinationthereof. For instance, memory may be any non-transitorycomputer-readable storage medium having computer-readable instructions(e.g., computer-readable program code instructions) stored thereof thatare executable by a processor.

The term “computing device” is used herein to refer to any one or all ofprogrammable logic controllers (PLCs), programmable automationcontrollers (PACs), industrial computers, desktop computers, personaldata assistants (PDAs), laptop computers, tablet computers, smart books,palm-top computers, personal computers, kiosk computers, smartphones,smartwatches, headsets, smart speakers, and similar electronic devicesequipped with at least a processor configured to perform the variousoperations described herein. Computing devices such as smartphones,laptop computers, tablet computers, headsets, and smartwatches aregenerally referred to herein as mobile devices.

The terms “server” and “server device” are used to refer to anycomputing device capable of functioning as a server, such as a masterexchange server, web server, mail server, document server, or any othertype of server. A server may be a dedicated computing device or acomputing device including a server module (e.g., an application whichmay cause the computing device to operate as a server). A server module(e.g., server application) may be a full function server module, or alight or secondary server module (e.g., light or secondary serverapplication) that is configured to provide synchronization servicesamong the dynamic databases on computing devices. A light server orsecondary server may be a slimmed-down version of server typefunctionality that can be implemented on a computing device, such as asmart phone, thereby enabling it to function as an Internet server(e.g., an enterprise e-mail server) only to the extent necessary toprovide the functionality described herein.

The terms “circuitry,” “module,” “utility,” and other similar termsshould be understood broadly to include hardware. In some embodiments,these terms may also include software for configuring the hardware. Forexample, in some embodiments, “circuitry” may include processingcircuitry, memory, communications circuitry, input-output circuitry,other circuitry, or a combination thereof. In some embodiments, otherelements of the present disclosure may provide or supplement thefunctionality of particular circuitry, modules, or utilities.

The term “client device” refers to computer hardware and/or softwarethat is configured to access a service made available by a server. Theserver is often (but not always) on another computer system, in whichcase the client device accesses the service by way of a network. Clientdevices may include, without limitation, smart phones, tablet computers,laptop computers, wearables, personal computers, enterprise computers,and the like.

As used herein, the terms “user,” “client,” “consumer,” and “customer”may be used interchangeably to refer, without limitation, to a client,customer, purchaser, shopper, user and the like who may be using aclient device to receive and interact with a device rendered object.

The term “device rendered object” may be used to refer to a set ofexecutable instructions that, when executed by a processor of acomputing device, renders an object for display. An example of a devicerendered object may be an instrument of a promotion. The instrument maybe any type of gift card, tender, electronic certificate, medium ofexchange, voucher, or the like that embodies terms of a promotion fromwhich the instrument resulted and may be used toward at least a portionof the purchase, acquisition, procurement, consumption or the like ofgoods, services and/or experiences. In some examples, the instrument maytake the form of tender that has a given value that is exchangeable forgoods, services and/or experiences and/or a reduction in a purchaseprice of a particular good, service or experience. In some examples, theinstrument may have multiple values, such as accepted value, apromotional value, and/or a residual value. In some examples, theaccepted value of the instrument is defined by the value exchanged forthe instrument. In some examples, the promotional value is defined bythe promotion from which the instrument resulted and is the value of theinstrument beyond the accepted value. In some examples, the residualvalue is the value after redemption, the value after the expiration orother violation of a redemption parameter, the return or exchange valueof the instrument and/or the like.

In embodiments, each device rendered object has associated dataindicating one or more categories (such as a restaurant promotion, a spapromotion, a travel promotion, a local promotion, etc.), sub-categories(such as a sushi restaurant), location, hyper-locations, prices or thelike. For example, an object may be associated with a gym, and the gymmay be associated with category data such as “beauty, wellness, andhealthcare,” “food and drink,” “leisure offers and activities” and/or“services.” A gym may also have more than one location, and may be ableto provide promotions in a variety of price ranges and time frames. Thedevice rendered object may be further associated with sub-category datasuch as “Chinese food” or “Mexican food.”

As used herein, the term “device rendered object service” may include aservice that is accessible via one or more computing devices and that isoperable to provide device rendered object services on behalf of one ormore providers that are offering one or more instruments that areredeemable for goods, services, experiences and/or the like. In someexamples, the device rendered object service may take the form of aredemption authority, a payment processor, a rewards provider, an entityin a financial network, a promoter, an agent and/or the like. As such,the device rendered object service is, in some example embodiments,configured to present one or more device rendered objects (e.g.,promotions) to client devices, accept payments for device renderedobjects from client devices, issue instruments upon acceptance of anobject, participate in redemption, generate rewards, provide a point ofsale device or service, issue payments to providers and/or otherwiseparticipate in the exchange of goods, services or experiences forcurrency, value and/or the like.

In the present embodiments, device rendered objects may have associatedtherewith one or more attributes. It should be understood that each ofthe attributes discussed in this application, may be in some embodimentsa primary or a secondary attribute. A list of example attributes isprovided below.

In some implementations, one attribute may represent category data,specifically beauty, wellness and healthcare. To determine a normalizedvalue of the one attribute, an algorithm may be executed on associatedraw data related to a device rendered object transaction. For example,if the category data associated with a device rendered objecttransaction is “beauty, wellness and healthcare,” the value of the oneattribute is 1. If the category data associated with a device renderedobject transaction is not “beauty, wellness and healthcare,” the valueof the one attribute is 0.

In some embodiments, a number of example attributes relate to categorydata and sub-category data. There is also a number of attributes relatedto location data. It should be understood that attribute data related tolocation and hyper location data may be generated in a similar fashion.For example, if the location data associated with a device renderedobject transaction is “Chicago,” the value of a corresponding attributemay be 1. If the category data associated with a device rendered objecttransaction is not “Chicago,” the value of the corresponding attributemay be 0.

It should also be understood that device rendered object transactionsmay have associated data indicating one or more categories,sub-categories, location, hyper-locations, prices or the like. Forexample, a device rendered object transaction may be associated with agym, and the gym may be associated with category data such as “beauty,wellness, and healthcare,” “Food and drink,” “Leisure Offers andActivities” and/or “Services.” A gym may also have more than onelocation, and may be able to provide promotions in a variety of priceranges and time frames.

Example attribute data may also be associated with review websites suchas Yelp®, Google®, Yahoo®, City Search®, Trip Advisor®. It should beunderstood that any review website could have associated attribute data,such as for example Zagat®, Bing® or the like. It should also beunderstood that attribute data associated with reviews related to one ormore categories may be generated. For example, a website may providereviews related to a provider's food, service, décor, price or the like.Attribute data may be generated related to each of one or morecategories, such as a provider's food, service, décor, price, or thelike.

It should also be understood that a location, hyper-location, categoryor sub-category may affect data available for the provider of apromotional transaction. For example, device rendered objecttransactions purchased from providers from different locations (e.g.,different countries, states, neighborhoods or the like) may beassociated with different attribute data. For example, differentcountries have different credit reporting rules and agencies. Thus, apromotion and marketing system may utilize a first combination ofattributes for a device rendered object transaction from a provider in afirst location, hyper-location, category or sub-category and a secondcombination of attributes for a device rendered object transaction fromprovider in a second location, hyper-location, category or sub-category.

In one implementation, other attribute data may be used. For example,attribute “bad month” may relate to identifying device rendered objecttransactions associated with booking months where prior device renderedobject transactions have involved products, services, and/or otherofferings in one or more categories categorized as having high returnrates based on prior performance data. The feature may be normalizedsuch that if the sub-category of the device rendered object transactionis associated with high return rates, the attribute is 1, else theattribute is 0. Other example attributes are “bad market,” which may benormalized such that if the device rendered object transaction requirestravel to a location and/or hyper local region associated with highreturn rates, the attribute is 1, else the attribute is 0. Anotherexample feature is “average price for sub-category.”

Similar attributes may be clustered, grouped, or aggregated. Forexample, attributes associated with locations or location basedattributes may be grouped under header attribute “location.” Forexample, a division attributes specifying a division where the promotionis offered and an attribute specifying a distance from a center of acity where a promotion is offered may be clustered under the locationheader attribute. Similarly, attributes associated with “pastperformance,” “category & service,” “completion,” “maximum capacity,”“hero score,” “review data” may each also be clustered and/or groupedunder header attributes. For example, with respect to device renderedobject transaction associated with a merchant providing a service,attributes specifying past performance data may be for (1) the samemerchant and the same service, (2) only the same merchant, (3) only thesame service may be clustered under attribute header “past performance.”Table A shows example attributes.

TABLE A Type Features Location division, distance to city, etc. pastperformance same merchant & service, same service, same merchant, etc.category & service service header, primary deal service, etc. bookingmonth holiday, pre-holiday, etc. maximum capacity units_avail, etc. heroscore predicted 30 day bookings, predicted 60 day bookings, etc. reviewdata yelp_n_review, yelp_rating, yelp_avil, google_n_review,facebook_fans, etc.

The clustered or the non-clustered attributes may be used to train amachine learning model. It should be understood that the selection ofattributes or clusters of attributes for training machine learningmodels may greatly affect the model performance. In someimplementations, attributes and/or clusters of attributes are selectedbased on statistical analysis. In some implementations, selection of themost significant attributes is based on one or more different attributeselection approaches. These approaches may be (1) forward selection,which is starting with the most significant attributes and incrementallyadding a next significant attribute until the model is stable; (2)backward elimination, which starts with all the attributes and excludethe non-significant attributes one by one until the model is stable; (3)a combination of forward selection and backward elimination; and (4)checking the significance of the attribute by statistical model(regression). In one embodiment, each attribute selection approach maygive a subset of significant attributes. The attributes that are notshown to be significant by one or more of the attribute selectionapproaches may be excluded from the model.

As used herein, the terms “provider” and “merchant” may be usedinterchangeably and may include, but are not limited to, a businessowner, consigner, shopkeeper, tradesperson, vendor, operator,entrepreneur, agent, dealer, organization or the like that is in thebusiness of providing a good, service or experience to a consumer,facilitating the provision of a good service or experience to a consumerand/or otherwise operating in the stream of commerce. The “provider” or“merchant” need not actually market a product or service via the devicerendered object service, as some merchants or providers may utilize thedevice rendered object service only for the purpose of gathering clientprofile information, vector activity information, similarity tableinformation, or the like.

The term “device rendered object interaction” refers to electronicinteraction with a device rendered object by a client device. In someexamples, the device rendered object interaction may take the form of anaccessing or selection of the device rendered object, a viewing of thedevice rendered object on an electronic display, a scrolling over thedevice rendered object, a retrieval of the device rendered object, apurchase transaction completed for the device rendered object, and/orthe like.

In some embodiments, the interaction data may include various otherinteractions, including without limitation, scroll-over (e.g.,mouse-over) events and durations, the amount of time spent by the clientdevice viewing particular content, the rate at which impressions ofparticular content result in sales associated with that content,demographic information associated with each particular client using theclient device, data indicating other content accessed by the clientdevice (e.g., browser cookie data), the time or date on which contentwas accessed, the frequency of impressions for particular content,associations between particular clients or client demographics andparticular impressions, and/or the like.

The term “device rendered object interaction signal” refers toelectronic indication that an interaction as defined above has occurredwith respect to a particular device rendered object.

It should be appreciated that the term “programmatically expected”indicates machine prediction of occurrence of certain events.

As used herein, the term “likelihood” refers to a measure of probabilityfor occurrence of a particular event. For example, the likelihood that aclient device will interact with a device rendered object may be a valueassociated with a specific scale. In some implementations, the machinepredictions discussed above and below are based, at least in part, onthe “likelihood” that an event will occur. Similarly, in someimplementations, machine predictions are based on attributes associatedwith a client profile and/or an associated device rendered object.

The term “R” refers to an open source programming language and softwareenvironment for statistical computing and graphics that is supported bythe R Foundation for Statistical Computing. The R language is widelyused among statisticians and data miners for developing statisticalsoftware and data analysis.

The term “Hadoop” refers to an open source software framework used fordistributed storage and processing of dataset of big data using theMapReduce programming model. Hadoop includes a storage part (HDFS) and aprocessing part which is the MapReduce programming model. The term“HDFS” refers to a storage part of the core of Hadoop, known as HadoopDistributed File System (HDFS). As referred to herein, “Gdoop” is amodified implementation of Hadoop.

Hadoop consists of computer clusters built from commodity hardware. Allthe modules in Hadoop are designed with a fundamental assumption thathardware failures are common occurrences and should be automaticallyhandled by the framework. Hadoop splits files into large blocks anddistributes them across nodes in a cluster. It then transfers packagedcode into nodes to process the data in parallel. This approach takesadvantage of data locality, where nodes manipulate the data they haveaccess to. This allows the dataset to be processed faster and moreefficiently than it would be in a more conventional supercomputerarchitecture that relies on a parallel file system where computation anddata are distributed via high-speed networking.

The term “machine learning model” refers to a machine learning task. Amachine learning model comprises a title (e.g., User Lifetime Value) andencompasses one or more target variables.

The term “target variable” refers to a value that a machine learningmodel is designed to predict. In the present embodiments, historicaldata is used to train a machine learning model to predict the targetvariable. Historical observations of the target variable are used forsuch training.

The term “machine learning model experiment” refers to a method forpredicting the target variables that comprise a machine learning model.The machine learning model experiment represents a certain set offeatures provided to a certain algorithm with a certain set ofhyper-parameters. A machine learning model experiment may haveassociated therewith a machine learning model experiment name and amachine learning model experiment description.

The term “machine learning model selection” refers to an electronicselection of a machine learning model available for inclusion in amachine learning model experiment. A machine learning model selectionmay be one or more of a touch screen input, mouse click or keyboardentry input provided to a computing device, and the machine learningmodel selection may be made from a displayed menu of several availablemachine learning models.

The term “machine learning model name” refers to a string of text and/ornumbers are used to identify a machine learning model.

The term “machine learning model experiment description” refers to aplurality of strings of text and/or numbers used to describe a design,notes, functionality, purpose, intention, or other information of amachine learning model experiment.

The term “machine learning model experiment creation input” refers toelectronic input provided to a computing device indicating andinitiating a creation of a machine learning model experiment.

The term “machine learning model experiment run” refers to an executionof a machine learning model experiment.

The term “machine learning model experiment run type” refers to the typeof experiment being run on a machine learning model. A machine learningmodel experiment run may be of a machine learning model experiment runtype, for example, a training run, a batch scoring run, and an always-onstreaming run.

The term “machine learning model score” refers to a particularprediction of a value of a target variable produced by the execution ofa machine learning model experiment.

Machine learning model execution includes two modules: the machinelearning task itself and the context in which the machine learning taskis executed. The machine learning task itself may be referred to as the“machine learning model engine” and the execution context may bereferred to as the “machine learning model execution engine.” Machinelearning engine models and machine learning model execution enginesrepresent an area of great volatility within the machine learning modellifecycle management context.

The term “machine learning model engine” refers to mechanisms fortraining a machine learning model experiment and producing a predictionusing the trained artifact. The machine learning model engineencompasses the particular algorithms, libraries, and methods fortraining and executing machine learning model tasks. Examples of amachine learning model engine include R, Foundry, or an implementationin Java. Machine learning model engines, according to the presentembodiments, deliver performance metrics generated during training runs.The performance metrics may be dependent upon the particular machinelearning model engine and a particular chosen algorithm employed by themachine learning model.

The term “machine learning model execution engine” refers to anexecution context in which a machine learning model engine is executed.Examples of a machine learning model execution engine include executinga PySpark job on Hadoop or executing a Java scoring job on a cluster.Other examples of machine learning model execution engines includeexercising a machine learning model within a MapReduce job over an HDFSfile, and exercising a machine learning model by feeding tuples readfrom a Kafka stream. Another example of a machine learning modelexecution engine includes exposing a machine learning model via a RESTendpoint, allowing for ad-hoc, on demand scoring. In embodiments, when amachine learning model is scored by a machine learning model executionengine, the scores produced by the machine learning model are collectedand compared to later real observations of the target variable toprovide metrics on machine learning model performance. By comparingpredictions (or scores) to actual measurements, the present embodimentsprovide error and correlation metrics.

The term “machine learning model engine selection” refers to anelectronic selection of a machine learning model engine for executing amachine learning model experiment. A machine learning model engineselection may be one or more of a touch screen input, mouse click orkeyboard entry input provided to a computing device.

The term “machine learning model experiment publishing selection” refersto an electronic selection of an option to publish a machine learningmodel experiment and associated data. A machine learning modelexperiment publishing selection may be one or more of a touch screeninput, mouse click or keyboard entry input provided to a computingdevice.

The term “machine learning model execution environment” refers to anenvironment providing the execution context, or machine learning modelexecution engine, for executing a machine learning model engine.

The term “dataset” refers to a collection of data. A dataset maycorrespond to the contents of a single database table, or a singlestatistical data matrix, where every column of the table represents aparticular variable, and each row corresponds to a given member of thedataset in question. The dataset may be comprised of tuples.

Inherent to every machine learning model is the schema of the datasetthe machine learning model expects. In the present system, a dataset isdefined separate from a machine learning model engine associated with amachine learning model experiment. The dataset is delivered to themachine learning model engine by the machine learning model executionengine. This enables the same machine learning model experiment sourcecode to operate in a batch mode, a streaming mode, or a real-time mode.

In the present embodiments, dataset generation is decoupled from machinelearning model execution. This enables reuse of generated datasets bydifferent machine learning model experiments and by different machinelearning models.

The terms “user interface” and “UI” refer to a programmatic display forenabling the interaction between humans and a computing device. The goalof user interface design is to produce a user interface which makes iteasy (e.g., self-explanatory), efficient, and enjoyable (e.g.,user-friendly) to operate a computing device in a way which produces adesired result. This generally means that the operator needs to provideminimal input to achieve the desired output, and also that the computingdevice minimizes undesired outputs to the human. In the presentembodiments, performance metrics may be displayed using a userinterface. In addition, scoring statistics may be gathered and whenmachine learning model performance crosses a particular threshold analert may be displayed on the user interface. In the presentembodiments, a user interface may provide visual display of machinelearning model execution run status in real-time.

The term “API” refers to an Application Programming Interface (API),which is a set of subroutine definitions, protocols, and tools forbuilding application software. An API is a set of clearly definedmethods of communication between various software components. An API maybe for a web-based system, operating system, database system, computerhardware or software library. An API specification may take many forms,but often includes specifications for routines, data structures, objectclasses, variables or remote calls. POSIX, Microsoft Windows API, theC++ Standard Template Library and Java APIs are examples of differentforms of APIs.

The terms “feature” and “features” refer to aggregations of raw datarepresenting device rendered object interactions (e.g., transactions,clicks, computing device engagement) rolled up to an entity and combinedwith attributes of the entity. In the present embodiments, supervisedlearning tasks use features. In the present embodiments, each featurehas associated therewith a “feature identifier” comprising text and/ornumbers for uniquely identifying the feature.

The term “feature catalog” refers to a digital repository that maintainsdefinitions of features and datasets.

The term “feature vector” refers to a vector comprising a plurality offeatures. In the present embodiments, values in a feature vector areobserved some time X before a value of the target variable was observed.The value X may be a standard offset or it may be dynamically generatedbased on when the target variable was observed.

The term “DaaS” refers to data as a service. DaaS builds on the conceptthat data may be provided on demand.

The terms “model parameter” and “model parameters” refer to propertiesof data that is a part of the training dataset that are learned duringtraining by a model classifier or other machine learning model. Forexample in case of a natural language processing (NLP) task: wordfrequency, sentence length, noun or verb distribution per sentence, thenumber of specific character n-grams per word, lexical diversity, etc.In the present embodiments, model parameters differ for each experimentand depend on the type of data and task at hand.

The terms “model hyper-parameter” and “model hyper-parameters” refer toproperties of data that is part of the training dataset that are commonfor similar machine learning models and cannot be learned duringtraining but are established or defined beforehand. A typical set ofhyper-parameters for neural networks includes the number and size of thehidden layers, weight initialization scheme, learning rate and itsdecay, dropout and gradient clipping threshold, etc.

The term “simultaneously” refers to a relation between two eventshappening at relatively the same time during a time frame of reference.

The terms “REST API” and “RESTful API” refer to web service APIs thatadhere to the REST architectural constraints. HTTP-based RESTful APIsare defined with the following aspects: base URL, such ashttp://api.example.com/resources/; an internet media type that definesstate transition data elements (tells the client how to compose requestsfor transitions to all the next available application states; could beas simple as a URL or as complex as a Java applet); and standard HTTPmethods (e.g., OPTIONS, GET, PUT, POST, and DELETE).

The term “performance metrics” refers to collected data and comparisonsrepresenting behavior of a machine learning model. For example, theperformance metrics of a machine learning model may provide a measure ofa level of accuracy of the predictions provided by the machine learningmodel.

The term “machine learning model scoring data file” refers to a filecontaining a trained machine learning model that is to be scored usingembodiments of the present disclosure.

The term “machine learning model scoring file” refers to a script thatwhen executed scores a trained machine learning model contained in amachine learning model scoring data file and provides an output dataframe containing results of the scoring.

The terms “input data” and “input data frame” refer to a set of datathat is used for scoring a trained machine learning model.

The term “memory location” refers to a location in a digital repository.

The term “machine learning model input data path” refers to a memorylocation at which machine learning model input data is stored andaccessible.

The term “machine learning model data frame” refers to data output froma scoring operation.

The term “observation timestamp” refers to a captured instance ofnetwork time associated with when an observation was made and stored.

The term “score timestamp” refers to a captured instance of network timeassociated with when scored was calculated/generated and stored

The terms “observation” and “observations” refer to an acquisition of aninstance value of a variable at a given point in time (e.g., networktime).

The term “recent observation” refers to the last observation of avariable that was captured and stored.

The term “historical observation” refers to past observations of avariable that were captured and stored. For example, an observation thatwas captured before the most recent observation may be referred to as ahistorical observation.

The tem “associated observations” refers to observations of variablesother than the target variable that are captured at the same time as aparticular observation of the target variable

The term “non-target variables” refers to variables other than thetarget variable.

The term “access” refers to the ability to retrieve any piece of dataelectronically from a digital repository.

It should be appreciated that the term “subset” describes a propersubset. A proper subset of set is portion of the set that is not equalto the set. For example, if elements A, B, and C belong to a first set,a subset including elements A and B is a proper subset of the first set.However, a subset including elements A, B, and C is not a proper subsetof the first set.

Having set forth a series of definitions called-upon throughout thisapplication, an example system architecture is described below forimplementing example embodiments and features of the present disclosure.

System Architecture

Methods, systems, apparatuses, and computer program products of thepresent disclosure may be embodied by any of a variety of devices. Forexample, the system, apparatus, method, and computer program product ofan example embodiment may be embodied by a networked device, such as oneor more servers, remote servers, cloud-based servers (e.g., cloudutilities), or other network entities, configured to communicate withone or more devices, such as one or more data storage devices, userdevices, or a combination thereof. Example embodiments of the userdevices include any of a variety of stationary or mobile computingdevices, such as a smartphone, laptop computer, tablet computer, desktopcomputer, electronic workstation, or any combination of theaforementioned devices.

FIGS. 1A-1D illustrate example environments of example systems ofexample sets of devices and circuitries that may be involved in someexample embodiments described herein. In this regard, FIG. 1A disclosesan example environment 100 within which embodiments of the presentdisclosure may operate to manage machine learning model lifecycles. Asillustrated, a machine learning model lifecycle management system 102may be connected to one or more server devices 104 in communication withone or more databases 106. The machine learning model lifecyclemanagement system 102 may be connected to one or more user devices 110,one or more programmatic client devices 112, one or more remote serverdevices 114, and one or more remote data storage devices 116 through oneor more communications networks 108. In some embodiments, one or morecommunications networks 108 may comprise the Internet; Ethernet; awireless local area network (LAN); a virtual private network (VPN); acellular network; a satellite network; a proximity-based network; awireless personal area network (PAN), any other suitable communicationsnetwork; or any combination thereof. In some embodiments, the machinelearning model lifecycle management system 102 may be configured tomanage the lifecycles of machine learning models as described in furtherdetail below.

The machine learning model lifecycle management system 102 may beembodied as one or more circuitries, computers, or computing systems asdescribed herein. The one or more server devices 104 may be embodied asone or more servers, remote servers, cloud-based servers (e.g., cloudutilities), circuitries, processors, or any other suitable serverdevices, or any combination thereof. The one or more server devices 104receive, process, generate, and transmit data, signals, content, andelectronic information to facilitate the operations of the machinelearning model lifecycle management system 102.

The one or more databases 106 may be embodied as one or more datastorage devices, such as a Network Attached Storage (NAS) device ordevices, or as one or more separate databases or servers. In someembodiments, the one or more databases 106 may be configured to provideDaaS. The one or more databases 106 include information accessed andstored by the machine learning model lifecycle management system 102 tofacilitate the operations of the machine learning model lifecyclemanagement system 102. For example, the one or more databases 106 maystore machine learning model selections, machine learning modelexperiment creation input (e.g., comprising, but not limited to, machinelearning model experiment names, machine learning model experimentdescriptions, machine learning model execution engine selections,machine learning model scoring files, and machine learning model scoringdata files), machine learning model experiment run types (e.g., batchscoring, real-time scoring, training), machine learning model input datapaths, machine learning model scores, machine learning model statistics(e.g., comprising, but not limited to, feature identifiers, scoretimestamps, scores, and experiment identifiers), machine learning modelexperiment publishing selections, machine learning model trainingdatasets (e.g., comprising, but not limited to, observations of targetvariables, and feature vectors comprising associated observations ofnon-target variables observed at times relative to the targetvariables), feature catalogs (e.g., comprising, but not limited to,feature definitions), UI data and content, machine learning modelprediction data, any other suitable data or electronic information, orany combination thereof. In some embodiments, the one or more databases106 may store user account credentials for users of one or more userdevices 110A-110N, one or more programmatic client devices 112A-112N,remote server devices 114, remote data storage devices 116, or acombination thereof. In some embodiments, the one or more databases 106may store data regarding device characteristics of one or more userdevices 110A-110N, one or more programmatic client devices 112A-112N,remote server devices 114, remote data storage devices 116, or acombination thereof.

The one or more user devices 110A-110N may be embodied by one or morecomputing devices, such as one or more “human clients” (e.g., clientdevices operable by humans). Information received by the machinelearning model lifecycle management system 102 from the one or more userdevices 110A-110N may be provided in various forms and via variousmethods. For example, the one or more user devices 110A-110N may belaptop computers, smartphones, netbooks, tablet computers, wearabledevices, desktop computers, electronic workstations, or the like, andthe information may be provided through various modes of datatransmission provided by these user devices. In some embodiments, usersmay use one or more user devices 110A-110N to input data and generateinteraction data. The machine learning model lifecycle management system102, one or more remote server devices 114, or a combination thereof mayreceive the data, or electronic information indicative of the data,directly or indirectly from the one or more user devices 110A-110N. Themachine learning model lifecycle management system 102, one or moreremote server devices 114, or a combination thereof may transmit UIdata, or electronic information indicative of the UI data, directly orindirectly to the one or more user devices 110A-110N for display on adisplay screen communicatively connected to the one or more user devices110A-110N. In some embodiments, users may use one or more user devices110A-110N to display UI data, input data, and facilitate otheroperations of the machine learning model lifecycle management system102.

The one or more programmatic client devices 112A-112N may be embodied byone or more computing devices, servers, or cloud utilities. Informationreceived by the machine learning model lifecycle management system 102from the one or more programmatic client devices 112A-112N may beprovided in various forms and via various methods. For example, the oneor more programmatic client devices 112A-112N may be servers, cloudutilities, laptop computers, smartphones, netbooks, tablet computers,wearable devices, desktop computers, electronic workstations, or thelike, and the information may be provided through various modes of datatransmission provided by these user devices. In some embodiments, one ormore programmatic client devices 112A-112N may be configured to receive,generate, store, and transmit data as a client service. The machinelearning model lifecycle management system 102, one or more remoteserver devices 114, or a combination thereof may receive data directlyor indirectly from, or transmit data directly or indirectly to, the oneor more programmatic client devices 112A-112N.

In embodiments where a user device 110A-110N or a programmatic clientdevice 112A-112N is a mobile device, such as a smartphone or tablet, themobile device may execute an “app” (e.g., a thin-client application) tointeract with the machine learning model lifecycle management system102, the one or more remote servers, the one or more remote data storagedevices 116, or a combination thereof. Such apps are typically designedto execute on mobile devices, such as tablets or smartphones. Forexample, an app may be provided that executes on mobile device operatingsystems such as Apple Inc.'s iOS, Google LLC's Android®, or MicrosoftCorporation's Windows®. These platforms typically provide frameworksthat allow apps to communicate with one another and with particularhardware and software components of mobile devices. For example, themobile operating systems named above each provide frameworks forinteracting with location services circuitry, wired and wireless networkinterfaces, user contacts, and other applications in a manner thatallows for improved interactions between apps while also preserving theprivacy and security of individual users. In some embodiments, a mobileoperating system may also provide for improved communications interfacesfor interacting with external devices (e.g., user devices, programmaticclient devices, data storage devices). Communication with hardware andsoftware modules executing outside of the app is typically provided viaAPIs provided by the mobile device operating system.

The one or more remote server devices 114 may be embodied by anysuitable computing device. In some embodiments, the one or more remoteserver devices 114 may be embodied as one or more servers, remoteservers, cloud-based servers (e.g., cloud utilities), processors, or anyother suitable devices, or any combination thereof. In some embodiments,the one or more remote server devices 114 may be embodied as one or moreGdoop processing devices, such as one or more Spark clusters or Gdoopresource managers. In some embodiments, the one or more remote serverdevices 114 may receive, process, store, generate, and transmit data,signals, and electronic information to facilitate the operations of themachine learning model lifecycle management system 102. Informationreceived by the machine learning model lifecycle management system 102from one or more remote server devices 114 may be provided in variousforms and via various methods. It will be understood, however, that insome embodiments, the one or more remote server devices 114 need notthemselves be servers, but may be peripheral devices communicativelycoupled to servers.

The one or more remote data storage devices 116 may be embodied by anysuitable computing device. In some embodiments, the one or more remotedata storage devices 116 may be embodied as one or more NAS devices, oras one or more separate databases or database servers. In someembodiments, the one or more remote data storage devices 116 may beembodied as one or more Gdoop storage devices, such as HDFS. In someembodiments, the one or more remote data storage devices 116 mayreceive, process, store, generate, and transmit data, signals, andelectronic information to facilitate the operations of the machinelearning model lifecycle management system 102. For example, the one ormore remote data storage devices 116 may store artifacts (e.g., trainedartifacts, execution artifacts). Information received by the machinelearning model lifecycle management system 102 from one or more remotedata storage devices 116 may be provided in various forms and viavarious methods. It will be understood, however, that in someembodiments, the one or more remote data storage devices 116 need notthemselves be databases or database servers, but may be peripheraldevices communicatively coupled to databases or database servers.

Additionally or alternatively, the one or more user devices 110A-110N,the one or more programmatic client devices 112A-112N, the one or moreremote server devices 114, the one or more remote data storage devices116, or any combination thereof may interact with the machine learningmodel lifecycle management system 102 over one or more communicationsnetworks 108. As yet another example, the one or more user devices110A-110N, the one or more programmatic client devices 112A-112N, theone or more remote server devices 114, the one or more remote datastorage devices 116, or any combination thereof may include varioushardware or firmware designed to interface with the machine learningmodel lifecycle management system 102. For example, an example userdevice 110A may be a user's laptop and may have an application storedthereon facilitating communication with the machine learning modellifecycle management system 102, whereas another example user device110B may be a user's smartphone and may have another application storedthereon facilitating communication with the machine learning modellifecycle management system 102. As yet another example, an exampleprogrammatic client device 112A may be a data scientist's computingdevice and may have an application stored thereon facilitatingcommunication with the machine learning model lifecycle managementsystem 102, whereas another example programmatic client device 112A-112Nmay be a purpose-built device offered for the primary purpose ofcommunicating with the machine learning model lifecycle managementsystem 102. As yet another example, a plurality of example remote serverdevices 114 and example remote data storage devices 116 may operatetogether as a Hadoop cluster configured to communicate with the machinelearning model lifecycle management system 102.

FIG. 1B illustrates an example logical design diagram of an exampleenvironment 120 comprising a machine learning model lifecycle managementsystem 102 in accordance with some example embodiments. Machine learningmodel lifecycle management system 102 enables handling, through a FluxAPI 124, incoming requests from one or more client devices 122 (e.g.,one or more programmatic client devices 112, one or more user devices110, or a combination thereof). One or more client devices 122 mayaccess machine learning model lifecycle management system 102 via acommunications network (not shown) (e.g., one or more communicationsnetworks 108, the Internet, LAN, WAN, or the like) using computingdevices. A user device 110 may interact with Flux API 124 through a FluxUI 128. The machine learning model lifecycle management system 102 maybe configured to receive requests (e.g., comprising input data) fromclient devices 122, generate corresponding machine learning modellifecycle records based on the received requests, and store thegenerated machine learning model lifecycle records in DaaS 126.Accordingly, embodiments of the present disclosure enable creation ofcorresponding records of a machine learning model lifecycle (modelsthemselves, experiments around the models, and executions of theexperiments) that result from handling the incoming requests.

In some embodiments, example environment 120 may comprise one or moreremote devices 130 comprising Gdoop Spark server 132 and Gdoop HDFS 134.Flux API 124 may access one or more remote devices 130 via acommunications network (not shown) (e.g., one or more communicationsnetworks 108, the Internet, LAN, WAN, or the like).

In some embodiments, example environment 120 may comprise multipleexecution engines configured to invoke the training or scoring with agiven set of machine learning model parameters on a specific technologystack, such as R models on Gdoop Spark server 132 or Scikit on SparkStreaming, and the like. Execution artifacts that specify executionparameters (R scripts or machine learning model scoring files needed forrunning a machine learning model, R trained model data file or machinelearning scoring data files, etc.) are uploaded from the client devices122 via Flux API 124 and stored at Gdoop HDFS 134 and the memorylocation of the artifacts in Gdoop HDFS 134 is stored as metadata inDaaS 126. Alternatively, artifacts may be stored in a database (notshown) and uploaded to Gdoop HDFS 134 as execution is about to happen.

Flux API 124, execution engines (e.g., Gdoop Spark server 132), andassociated processing modules may be implemented as embodied as one ormore circuitries, processors, memories, computers, or a combinationthereof. The Gdoop Spark server 132 may provide for receiving ofelectronic data from various sources, including but not necessarilylimited to the client devices 122. Example environment 120 may furthercomprise a database (not shown) to facilitate the operations of themachine learning model lifecycle management system 102. The database maybe embodied as one or more databases or data storage devices, such asone or more databases 106, remote data storage devices 116, NetworkAttached Storage (NAS) device, or a combination thereof, or as aseparate database server or servers. The database may compriseinformation accessed and stored by the Flux API 124, Flux UI 128, GdoopSpark server 132, or any other suitable device or circuitry tofacilitate the operations of the machine learning model lifecyclemanagement system 102.

FIG. 1B illustrates an example design diagram of an example environment140 comprising one or more client devices 122, one or more loadbalancers 144, one or more Flux API hosts 142, one or more Flux utilityhosts 148, DaaS 126, Gdoop Resource Manager 146, and Gdoop HDFS 134 inaccordance with some example embodiments. Flux API hosts 142 maycomprise Flux API 124. Flux utility hosts 148 may comprise a scheduler150, one or more Flux workers 152, and a job monitor 154.

In some embodiments, the machine learning model lifecycle managementsystem uploads execution artifacts that specify execution parameters (Rscripts or machine learning model scoring files needed for running amachine learning model, R trained model data file or machine learningscoring data files, etc.) from the one or more client devices 122 viaFlux API 124 and stores the execution artifacts at Gdoop HDFS 134. Themachine learning model lifecycle management system may store the memorylocation of the execution artifacts in Gdoop HDFS 134 as metadata inDaaS 126. Alternatively, machine learning model lifecycle managementsystem may store the execution artifacts in a database (not shown) andupload the execution artifacts to Gdoop HDFS 134 as execution is aboutto happen.

In some embodiments, the machine learning model lifecycle managementsystem uploads trained artifacts (e.g., the trained model artifacts,such as R model binary, that are generated from Flux worker 152 runs) toGdoop HDFS 134, indexed by a corresponding machine learning modelexecution identifier. Client devices 122 may later look up a machinelearning model, then a desired machine learning model experiment(associated with the machine learning model), then a desired machinelearning model experiment run instance (associated with the machinelearning model experiment) and retrieve the dataset location of theoutput. It will be appreciated that a machine learning model experimentmay have multiple runs by varying the score input set; in someinstances, it will be up to the client to find the correct run instanceto consume the result from.

In some embodiments, Flux API hosts 142 provides APIs (e.g., Flux API124) to manage machine learning model lifecycle to client devices overREST APIs. For example, fronted by a load balancer 144, Flux API 124 maycommunicate with DaaS 126 to store the internal data about the requestsreceived around machine learning model lifecycle management. In someembodiments, Flux API 124 may submit run jobs directly to Gdoop resourcemanager 146. Execution status is able to be queried via a REST API tofind out the completion status of the request.

In some embodiments, Flux workers 152 submit jobs to Gdoop resourcemanager 146, leaving the Flux API 124 to orchestrate the incoming modellifecycle management requests to suitable work units to be carried outby the Flux workers 152.

In some embodiments, job monitor 154 may periodically connect to DaaS126 to retrieve a list of jobs that are currently in RUNNING state andretrieve their corresponding Hadoop application ID handle. Job monitor154 may then connect to Gdoop resource manager 146 to retrieve the stateof the Gdoop execution and update the state in DaaS 126 directly.

In some embodiments, a scheduler 150 may submit jobs to Flux API 124based on a schedule defined to execute a machine learning modelexperiment on schedule.

In some embodiments, Flux workers 152 may pick-up runs that are in READYTO RUN states and launch the execution using the proper executionengines (e.g., in accordance with, in some instances, Table B below). Insome embodiments, this process may result in a cutoff of directcommunication between Flux API 124 and Gdoop resource manager 146.Instead, Flux API 124 writes the runs in states such that the Fluxworkers 152 may pick up these requests. The Flux workers 152 are lightweight meaning that the Flux workers 152 are configured to pick up thejob and launch respective backend processing on different executionengines (e.g., PySpark job on Gdoop or Java scoring job on cluster).Hence Flux workers 152 may be run on the same hosts as the Flux APIhosts 142. However, it may be preferable to keep the process spaceseparated between the Flux API 124 and the Flux workers 152, so that theFlux workers 152 do not slow down the Flux API 124.

FIG. 1D illustrates an example system architecture of an exampleenvironment 160 comprising machine learning model lifecycle managementcircuitry 162, batch execution circuitry 164, real-time executioncircuitry 166, archive 168, user device 170A, user device 170B (whichmay be the same as, or different from, user device 170A), client service172A (e.g., a client device executing a client service), and clientservice 172B (which may be the same as, or different from, clientservice 172A). In some embodiments, the machine learning model lifecyclemanagement system disclosed herein (e.g., machine learning modellifecycle management system 102) may comprise machine learning modellifecycle management circuitry 162, batch execution circuitry 164,real-time execution circuitry 166. In some embodiments, the machinelearning model lifecycle management system disclosed herein may furthercomprise, or be in communication with, archive 168. Archive 168 andartifact store 176 may be embodied as any suitable memory, database,database server, or combination thereof.

In some embodiments, machine learning model lifecycle managementcircuitry 162 may comprise Flux API 174 and artifact store 176. In someembodiments, machine learning model lifecycle management circuitry 162may further comprise Flux UI 178 communicatively coupled with userdevice 170A and Flux API 174, whereas the client service 172A may becommunicatively coupled directly with Flux API 174. In some embodiments,batch execution circuitry 164 may comprise one or more batch executionengines 180 and one or more Hadoop/Spark servers 182. In someembodiments, real-time execution circuitry 166 may comprise one or moreFoundry execution engines 184, one or more H2O execution engines 186,one or more R execution engines 188, and a gateway server 188communicatively coupled with user device 170B, client service 172B, orboth. In some embodiments, clients (e.g., user device 170A, clientservice 172A) may make requests to the machine learning model lifecyclemanagement circuitry 162 through the Flux API 174. Depending on the usecase, models may be executed by either the batch execution circuitry 164or the real-time execution circuitry 166.

In some embodiments, the machine learning model lifecycle managementcircuitry 162 may provide for automatic hyper-parameter tuning, whereclients may define a set of hyper-parameter ranges for the specificmodel algorithm. The machine learning model lifecycle managementcircuitry 162 may explore (e.g., analyze) the parameter space and pick(e.g., select) the set of hyper-parameters that yields optimal modelperformance. In some embodiments, the machine learning model lifecyclemanagement circuitry 162 may provide for automated model selection wheremultiple candidate implemented with different algorithms are trainedwith the same training data, and run alongside with the incumbent model.The machine learning model lifecycle management circuitry 162 maymonitor the performance of all candidates, and promote a candidate onceit out-performs the incumbent model. Accordingly, the machine learningmodel lifecycle management circuitry 162 may pick the algorithm that hasoptimal performance.

In some embodiments, the real-time execution circuitry 166 maycontainerize the microservice created for each model, where a containeris spun up when requested or when training finishes. In someembodiments, the real-time execution circuitry 166 may utilize Dockerfor easy provisioning of real-time scoring services, where a containeris spun up within an environment such as an Apache Mesos cluster whenrequested or when training finishes. This way the microservices are evenmore light-weight, and the overhead of box provisioning and modeldeployment is significantly reduced.

Accordingly, the machine learning model lifecycle management systemdescribed herein (e.g., machine learning model lifecycle managementsystem 102) is configured to manage the lifecycles of machine learningmodels and their execution, and to function as the interface positionedbetween two sets of users (e.g., a data science team and an engineeringteam). The machine learning model lifecycle management system providesfunctionalities to create Models and Experiments, execute models in bothbatch and real-time mode, and archive the predictions for AB testing andanalytical purposes. In some instances, the design of the machinelearning model lifecycle management system balances highly optimized,fast code with easy management. In one example, data scientist teams mayuse the machine learning model lifecycle management system via auser-friendly Flux UI (e.g., Flux UI 128, Flux UI 178). In anotherexample, engineering teams may directly invoke the REST APIs (e.g., FluxAPI 124, Flux API 174) programmatically.

As described above, in some embodiments, the machine learning modellifecycle management system may comprise three components: a centrallifecycle management component (e.g., machine learning model lifecyclemanagement circuitry 162, machine learning model lifecycle managementcircuitry 212); a batch execution engine (e.g., batch executioncircuitry 164, batch execution circuitry 214) that, in some instances,runs on Hadoop YARN; and a real-time execution engine (e.g., real-timeexecution circuitry 166, real-time execution circuitry 216) that, insome instances, enables utilizing models as a microservice.

In some embodiments, the lifecycle management component is configured tohandle incoming requests from clients to manage model lifecycle overREST APIs and orchestrate model execution. The lifecycle managementcomponent is configured to store data about Models, Experiments, andRuns in a database. The lifecycle management component is configured tolaunch model execution using the proper Execution Engines. The lifecyclemanagement component is further configured to persist information on thestatus of the model execution and to make that information available viaREST API. In some embodiments, the lifecycle management component is thecore of the machine learning model lifecycle management system and maybe only loosely coupled with other components, such as the executioncomponents. In some instances, new Execution Engines may be implementedand plugged into the system. In some embodiments, the executioncomponents may scale horizontally by adding more resources: more nodesfor batch execution; and more hosts for real-time execution.

In some embodiments, the batch execution component is configured tosubmit model execution to a Hadoop YARN cluster and parallelize thecomputation to achieve high-throughput.

In some embodiments, the real-time execution component is configured toexpose a trained machine learning model as a RESTful microservice toprovide predictions for incoming requests. The real-time executioncomponent is configured to serve as the execution engine for models thatneed on-demand, real-time scoring.

In some embodiments, the machine learning model lifecycle managementsystem provides a core library that encapsulates many runtimefunctionalities of the platform, which may be embedded in otherarchitectures. In some embodiments, clients may choose to use themachine learning model lifecycle management system as a service, orintegrate the runtime library into their own systems.

As a foundation for some embodiments, the machine learning modellifecycle management system may be discussed with reference to anexample system architecture comprising models, experiments, runs, runtypes, artifacts, model engines, execution engines, and input data.

A “Model” is an abstraction over specific machine learning tasks. AModel defines the target variable (e.g., what to predict). For example,a lead scoring model predicts the score of potential partners, and acustomer lifetime value model predicts the value of each customer to acompany. A model may serve as an organizational tool to differentiterations of machine learning models that attempt to address the sameor substantially similar problem. A model may have other attributes suchas the name of the field that uniquely identifies each feature vector(e.g., identifier).

Each Model may have a number of Experiments. An “Experiment” defines theconcrete plan of how to predict the target variable, including thespecific machine learning algorithm, the actual implementation of thealgorithm, and a set of hyper-parameters. Each Model may also haveexactly one Published Experiment, which serves as the default Experimentfor a given Model until a new Experiment is published. In this way afirst set of users (e.g., an engineering team) may be locked in with acertain Experiment and its defined schema of the feature vector, while asecond set of users (e.g., a data science team) may vary the Experimentsseparately without affecting the production environment.

For each Experiment, the client may trigger multiple Runs. A “Run”specifies an input dataset and has a certain Type, such as Training,Batch Scoring, Scoring, and Real-time Scoring. Training Runs generatenew model artifacts using a training set. Batch Scoring Runs producepredictions offline over a large batch of inputs. Scoring Runs producepredictions given a scoring set. Real-time Scoring Runs provision amicroservice that hosts the model for online prediction requests.

A model “Artifact” is the product of a training process that may beexecuted on a scoring dataset to provide scores. A model Artifact mayinclude serialized trained models (e.g., predictors) and associated sidedata such as the location of the Artifact within the artifact store.

A “Model Engine” is the machine learning library that providesimplementation of various algorithms, usually associated with certainprogramming languages. The machine learning model lifecycle managementsystem supports at least three Model Engines: R; Cognitive Foundry, aJava library; and H2O. New Model Engines may be integrated easily byimplementing a set of interfaces.

The machine learning model lifecycle management system 102 may compriseone or more Execution Engines. In some embodiments, the combination ofRun Types and Model Engines produce the Execution Engines, which willinvoke the training or scoring with a given set of model parameters onspecific technology stack, as shown in Table B:

TABLE B Model Type Run Type R Foundry H2O Batch Scoring R on PySparkSpark Spark Real-time Scoring Real-time R Real-time Foundry Real-timeH2O Training R on PySpark, or Foundry Training, H2O Training Single Rprocess or Single Java process

Depending on the execution engine, the machine learning model lifecyclemanagement system takes input data of different formats. For batchscoring and training, the machine learning model lifecycle managementsystem may take a path on HDFS where the input dataset is located. Thedataset may have various formats such as JSON, CSV, and TSV. In someinstances, the machine learning model lifecycle management system doesnot specify how the input data is generated, so the clients may use anytool to create the dataset. For example, some clients may use an ETLmanagement system, such as QED, to generate a scoring dataset. Forreal-time scoring, the machine learning model lifecycle managementsystem may take an HTTP payload in JSON.

In some embodiments, the machine learning model lifecycle managementsystem may provide a set of functionalities for the management of modellifecycle. In some embodiments, clients may create and delete Models,create new Experiments for Models, and create new Runs for an Experimentor for a Model where an Experiment is published. When creating a Model,the clients may provide a title and a description and one or more targetvariables this model will predict. In addition, the clients may specifyan identifier field that uniquely identifies each feature vector (e.g.,row) that will be scored. Unique identifiers are especially useful foroptimizing storage of model outputs for analytics.

In some embodiments, when creating an Experiment of a Model, clients mayspecify model parameters that include relevant data to make a Run ofspecified execution engine. For example, for R on PySpark executionengine, model parameters may include an R script that will be sourced toproduce the scores, an RData binary file that contains the serializedmodel object that will be loaded prior to the scoring, and anyadditional side data in the form of RData files. For the Foundryexecution engine, model parameters may contain the type of the model,random forest or logistic regression, and a set of hyper-parametersspecific to the model, such as the number of trees and maximum depth ofeach tree in a random forest, or the strength of L1 and L2regularization for logistic regression. Hyper-tuning may be implementedfor these models.

In one illustrative example, for a given Model, clients may publish anexisting Experiment, making it the default Experiment for this Model.This key feature allows the engineering team to “lock in” with onepublished Experiment and its associated feature vector schema, thusguaranteeing a stable production environment and eliminating thenecessity of modifying downstream clients of the Model for every smallchange in the Model. At the same time, data science teams mayindependently create new Experiments with different settings, and arethus able to iterate fast on newer versions of the Model. Once the datascientists are satisfied with the newer iteration and has communicatedchanges to the scoring process to the engineering team, either side cango ahead and publish the new Experiment, completing a lifecycle of oneparticular model implementation.

In some embodiments, when creating a Run, clients may specify the inputdata location. The machine learning model lifecycle management systemmay read input data from one or more HDFS clusters, and may take inputin multiple formats including line-delimited JSON, CSV, TSV, and othersuitable formats. Request to create a new batch-scoring or training Runfor a Model may receive a response containing the Run ID. Clients maythen check the status of the Run using the ID. For real-time scoring,the prediction is instantly provided as the response for the scoringrequest, and no status checking is needed.

In some embodiments, for an Experiment, Training Runs may producetrained Artifacts that store fitted model parameters and may be executedon a scoring dataset during scoring Runs to produce predictions.Artifacts may be stored at an artifact store, indexed by thecorresponding Run ID. Their locations may be preserved in the databaseand may be retrieved later for inspection. At the time of new Experimentcreation, Artifacts may be uploaded from the clients via the REST API.This feature may enable the data scientists to train a model offline andupload it to the machine learning model lifecycle management system onceit is ready for production. In some embodiments, at any moment, clientsmay use an API (e.g., Flux API) to look up the details of any Model,Experiment, and Run, and check the output predictions.

Various embodiments of the batch execution engine (e.g., batch executioncircuitry 164, batch execution circuitry 214) and the real-timeexecution engine (e.g., real-time execution circuitry 166, real-timeexecution circuitry 216) are described in further detail below.

With reference to the batch execution engine, in some embodiments, themachine learning model lifecycle management system may comprise severalexecution engines that operate in batch mode, R on PySparkscoring/training, and batch-scoring for Foundry and H2O. In someembodiments, the machine learning model lifecycle management system mayutilize Apache Spark for batch execution, and scaling-up may be done byrequesting more executors and more resources per executor. A Run that isexecuted by these engines may comprise multiple steps referred to asjobs. The machine learning model lifecycle management system may use asimple sequential workflow to manage the execution of these jobs. Jobtypes include Data Fetch, Execution, Monitor, and Archiving.

In some embodiments, when a request to create a Run is received, themachine learning model lifecycle management system may take the modelengine of the Experiment and Run type, and determine which executionengine to use. The machine learning model lifecycle management systemthen may fetch a predefined workflow template for the execution engine,and create entries in its job queue. Each Flux Worker box (e.g., one ormore Flux workers 152) will read the job queue and take one job at atime. Once an upstream job is successfully finished, the next job willbe picked up. Otherwise, the failed job will be retried till a maximumnumber of failures is reached, at which point the entire Run will failand clients will be notified.

In one illustrative example embodiment, a workflow may have thefollowing job types: Data Fetch, Execution, Monitor, and Archive:

Data Fetch. A Data Fetch Job is the first step in a workflow. It simplyreads the input data information of the Run and passes it to the nextjob.

Execution. The actual execution happens within the Execution Job. Themachine learning model lifecycle management system may take the modelengine of the Experiment and Run Type and decide which Execution Engineto use. For scoring Runs, the machine learning model lifecyclemanagement system executes the artifact on the input data using themodel Artifact and produces a scored dataset. For training Runs, themachine learning model lifecycle management system creates new modelArtifacts and, in some instances, uploads those new model Artifacts tothe artifact store and updates the Experiment accordingly.

Monitor. Continues to poll the status of the execution to check whetherit is finished (e.g., until it finishes), and whether it was successful.In some instances, Monitor may update the Run status.

Data Save (e.g., Archive). Archives predictions to HDFS and Teradata forlater use, typically for AB testing and offline analytics. If theexecution is successful, the machine learning model lifecycle managementsystem saves the output to different sinks. The machine learning modellifecycle management system may archive predictions to both HDFS andTeradata. Each Model may have its own table in both datastores, and rowsmay be indexed by Run ID and Experiment ID. Predictions may be indexedby Model ID, Experiment ID, and Run ID, so data scientists and analystscan exactly tell what algorithms and parameters lead to the scores. ForTraining Runs, the machine learning model lifecycle management systemmay save trained model Artifacts to the artifact store to be used byscoring Runs later.

In some embodiments, the machine learning model lifecycle managementsystem comprises a batch execution engine interface. In one example, forFoundry models, the machine learning model lifecycle management systemmay wrap around the underlying model objects with a Scorer interface,which has a score function that takes a payload and returns aprediction. In another example, for R models, the machine learning modellifecycle management system may launch a PySpark job on the Hadoop/Sparkcluster (e.g., Hadoop/Spark server 182). The Python process passes theinput data to the R runtime via a library that allows Python/Rinteroperation. The R runtime then parses the data into a dataframe andthen invokes a score function from an R library provided by the machinelearning model lifecycle management system. The Python process thenreads the serialized result from R and passes it back to Spark JVM.

One big challenge in the design process of the machine learning modellifecycle management system is how to ensure a uniform interface fordifferent Execution Engines. The Model Engines that the machine learningmodel lifecycle management system supports may be implemented withmultiple programming languages such as Java and R. Some of them may haveclearly defined APIs, making them easier to integrate, while others mayneed more careful design. For Foundry and H2O models, the machinelearning model lifecycle management system may wrap around theunderlying model objects with an interface that takes a payload andreturns a prediction. For R models, the machine learning model lifecyclemanagement system may comprise an adapter that runs arbitrary R scriptsparallelly in a Spark cluster. Within only a few constraints imposed bythe machine learning model lifecycle management system, an R script maybe executed in the same way that a data scientist does locally inRStudio, but for all the data partitions simultaneously. When a scoringRun starts, the machine learning model lifecycle management system maylaunch a PySpark job on the Spark cluster. For each partition, thePython process may pass the input data to the R runtime via a librarythat enables Python/R interoperation. The R runtime then may parse thedata into a dataframe, invoke a score function from an R libraryprovided by the machine learning model lifecycle management system, andpass the score back to the Python process. Subsequently, the PySpark jobmay write all the scored partitions into HDFS. The machine learningmodel lifecycle management system may use heuristics to optimize Sparkparameters such as number of partitions, and neither data scientists nordownstream clients of models on the machine learning model lifecyclemanagement system need to be aware of those details.

In some embodiments, the real-time execution engine may be configured toprovide clients with fast, on-demand scoring of a feature vector using amachine learning model. Flux Real-Time Scoring (RTS) is a component ofthe machine learning model lifecycle management system that encapsulatesa trained model behind a RESTful service. In some instances, RTS is acomponent of the machine learning model lifecycle management system thatprovides a RESTful service, which utilizes a trained model to providescores or predictions for incoming requests. Flux RTS serves as thereal-time execution engine for models on the machine learning modellifecycle management system. A single model Artifact may be hostedinside a web application, and used as a function to produce scores.Essentially, the machine learning model lifecycle management system, insome instances, may be configured to expose a machine learning model ina service-oriented manner to be used by other systems. In certaininstances, running a model as a microservice may the moststraightforward way to ensure that no unnecessary overhead costs arepaid.

In some embodiments, the Flux RTS service may distributed among multiplehosts, behind a Gateway server (e.g., gateway server 188) that acts asboth proxy and router. The Gateway server stores information on whatExperiment ID is associated with which hosts, and upon receiving aclient request to a given Experiment ID, the Gateway server routes thepayload to the corresponding hosts. In some instances, upon receivingresponse from the host, the Gateway server may simply proxy it back tothe client.

In some embodiments, Foundry and H2O workers may be responsible forscoring requests for models implemented with Foundry and H2O,respectively. In some embodiments where both the web application and themodel are implemented in Java, the web application may simply use themodel as an in-memory Java object to score. After the prediction ismade, Foundry and H2O workers may transmit the response back to theGateway server.

In some embodiments, for models implemented with R, the web applicationmay not directly use the model object as an in-memory function, andinstead use inter-process communication. Flux RTS may use a library thatallows JVM and R processes to communicate via a web socket. Uponreceiving the payload for a scoring request, the web application maytransmit the data to R via the socket, and invoke the score functionfrom the Flux R library. Once a prediction is made, the web applicationmay fetch the serialized result from R and transmit it back to theGateway server.

In some embodiments, the Model may be trained via Training Runs, anddeployed to Flux RTS hosts upon client request, in a manner similar topublishing an Experiment. Upon model deployment, Flux RTS will consultFlux REST APIs to get the current Artifact ID for the given Models, andload the artifact from the artifact store into memory. Even though themodel itself may be viewed as a fast function, the cost of loading andunloading a model instance is sometimes not negligible, so, in someinstances, model loading is done only once during deployment.

In some embodiments, the machine learning model lifecycle managementsystem 102 may provide for receiving a machine learning model selection,a machine learning model experiment creation input (e.g., a machinelearning model experiment name, a machine learning model experimentdescription, a machine learning model execution engine selection, atleast one machine learning model scoring file, and at least one machinelearning model scoring data file), a machine learning model experimentrun type (e.g., one of batch scoring, real-time scoring, or training),and a machine learning model input data path. The machine learning modellifecycle management system 102 may receive this data from varioussources, including but not necessarily limited to the one or more userdevices 110A-110N, the one or more programmatic client devices112A-112N, the one or more remote server devices 114, the one or moreremote data storage devices 116, or a combination thereof. In someembodiments, the machine learning model lifecycle management system 102may provide for determining a machine learning model execution enginebased on the machine learning model execution engine selection and themachine learning model experiment run type. In some embodiments, themachine learning model lifecycle management system 102 may provide forretrieving input data based on the machine learning model input datapath. The machine learning model lifecycle management system 102 mayretrieve the input data from various sources, including but notnecessarily limited to the one or more remote server devices 114, theone or more remote data storage devices 116, or a combination thereof.In some embodiments, the machine learning model lifecycle managementsystem 102 may provide for executing a machine learning model experimentbased on the machine learning model execution engine, the machinelearning model scoring file, the at least one machine learning modelscoring data file, and the input data. In some embodiments, the machinelearning model lifecycle management system 102 may provide forgenerating, based on the machine learning model experiment, one or moremachine learning model scores associated with the machine learning modelexperiment and the input data. In some embodiments, the machine learningmodel lifecycle management system 102 may provide for publishing the oneor more machine learning model scores.

Example Implementing Apparatus

The machine learning model lifecycle management system 102 describedwith reference to FIG. 1 may be embodied by one or more computingapparatuses, such as apparatus 200 shown in FIG. 2. As illustrated inFIG. 2, the apparatus 200 may include processing circuitry 202, memory204, input-output circuitry 206, communications circuitry 208,application programming interface (API) circuitry 210, machine learningmodel lifecycle management circuitry 212, batch execution circuitry 214,real-time execution circuitry 216, publishing circuitry 218, userinterface (UI) circuitry 220, and machine learning model trainingcircuitry 222. The apparatus 200 may be configured to execute theoperations described above with respect to FIG. 1 and below with respectto FIGS. 3-21. Although some of these components 202-222 are describedwith respect to their functional capabilities, it should be understoodthat the particular implementations necessarily include the use ofparticular hardware to implement such functional capabilities. It shouldalso be understood that certain of these components 202-222 may includesimilar or common hardware. For example, two sets of circuitry may bothleverage use of the same processor, network interface, storage medium,or the like to perform their associated functions, such that duplicatehardware is not required for each set of circuitry.

The use of the term “circuitry” as used herein with respect tocomponents of the apparatus 200 therefore includes particular hardwareconfigured to perform the functions associated with respective circuitrydescribed herein. Of course, while the term “circuitry” should beunderstood broadly to include hardware, in some embodiments, circuitrymay also include software for configuring the hardware. For example, insome embodiments, “circuitry” may include processing circuitry, storagemedia, network interfaces, input-output devices, and other components.In some embodiments, other elements of the apparatus 200 may provide orsupplement the functionality of particular circuitry. For example, theprocessing circuitry 202 may provide processing functionality, memory204 may provide storage functionality, and communications circuitry 208may provide network interface functionality, among other features.

In some embodiments, the processing circuitry 202 (and/or co-processoror any other processing circuitry assisting or otherwise associated withthe processor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204may be non-transitory and may include, for example, one or more volatileand/or non-volatile memories. For example, the memory 204 may be anelectronic storage device (e.g., a computer-readable storage medium). Inanother example, the memory 204 may be a non-transitorycomputer-readable storage medium storing computer-executable programcode instructions that, when executed by a computing system, cause thecomputing system to perform the various operations described herein. Thememory 204 may be configured to store information, data, content,applications, instructions (e.g., computer-executable program codeinstructions), or the like, for enabling the apparatus 200 to carry outvarious functions in accordance with example embodiments of the presentdisclosure. For example, the memory 204 may be configured to store:machine learning model selections; machine learning model experimentcreation input (e.g., comprising, but not limited to, machine learningmodel experiment names, machine learning model experiment descriptions,machine learning model execution engine selections, machine learningmodel scoring files, and machine learning model scoring data files);machine learning model experiment run types (e.g., batch scoring,real-time scoring, training); machine learning model input data paths;machine learning model scores; machine learning model statistics (e.g.,comprising, but not limited to, feature identifiers, score timestamps,scores, and experiment identifiers); machine learning model experimentpublishing selections; machine learning model training datasets (e.g.,comprising, but not limited to, observations of target variables, andfeature vectors comprising associated observations of non-targetvariables observed at times relative to the target variables); featurecatalogs (e.g., comprising, but not limited to, feature definitions); UIdata and content; machine learning model prediction data; any othersuitable data or data structures; or any combination or combinationsthereof. It will be understood that the memory 204 may be configured tostore partially or wholly any electronic information, data, datastructures, embodiments, examples, figures, processes, operations,techniques, algorithms, instructions, systems, apparatuses, methods, orcomputer program products described herein, or any combination thereof.

The processing circuitry 202 may be embodied in a number of differentways and may, for example, include one or more processing devicesconfigured to perform independently. Additionally or alternatively, theprocessing circuitry 202 may include one or more processors configuredin tandem via a bus to enable independent execution of instructions,pipelining, multithreading, or a combination thereof. The use of theterm “processing circuitry” may be understood to include a single coreprocessor, a multi-core processor, multiple processors internal to theapparatus, remote or “cloud” processors, or a combination thereof.

In an example embodiment, the processing circuitry 202 may be configuredto execute instructions stored in the memory 204 or otherwise accessibleto the processing circuitry 202. Alternatively or additionally, theprocessing circuitry 202 may be configured to execute hard-codedfunctionality. As such, whether configured by hardware or softwaremethods, or by a combination of hardware with software, the processingcircuitry 202 may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present disclosure while configured accordingly. As anotherexample, when the processing circuitry 202 is embodied as an executor ofprogram code instructions, the instructions may specifically configurethe processor to perform the operations described herein when theinstructions are executed.

In some embodiments, the apparatus 200 may include input-outputcircuitry 206 that may, in turn, be in communication with processingcircuitry 202 to provide output to the user and, in some embodiments, toreceive input such as a command provided by the user. The input-outputcircuitry 206 may comprise a user interface, such as a graphical userinterface (GUI), and may include a display that may include a Flux UI, aweb user interface, a GUI application, a mobile application, a clientdevice, or any other suitable hardware or software. In some embodiments,the input-output circuitry 206 may also include a keyboard, a mouse, ajoystick, a display device, a display screen, a touch screen, touchareas, soft keys, a microphone, a speaker, or other input-outputmechanisms. The processing circuitry 202, input-output circuitry 206(which may utilize the processing circuitry 202), or both may beconfigured to control one or more functions of one or more userinterface elements through computer-executable program code instructions(e.g., software, firmware) stored in a non-transitory computer-readablestorage medium (e.g., memory 204). Input-output circuitry 206 isoptional and, in some embodiments, the apparatus 200 may not includeinput-output circuitry. For example, where the apparatus 200 does notinteract directly with the user, the apparatus 200 may generate userinterface data for display by one or more other devices with which oneor more users directly interact and transmit the generated userinterface data to one or more of those devices. For example, theapparatus 200, using UI circuitry 220, may generate UI data (e.g., FluxUI data) for display by one or more display devices and transmit thegenerated UI data to those display devices.

The communications circuitry 208 may be any device or circuitry embodiedin either hardware or a combination of hardware and software that isconfigured to receive or transmit data from or to a network (e.g., oneor more communications networks 108) or any other device, circuitry, ormodule in communication with the apparatus 200. In this regard, thecommunications circuitry 208 may include, for example, a networkinterface for enabling communications with a wired or wirelesscommunication network. For example, the communications circuitry 208 mayinclude one or more network interface cards, antennae, buses, switches,routers, modems, gateway servers, load balancers, and supportinghardware and/or software, or any other device suitable for enablingcommunications via a network. In some embodiments, the communicationsinterface may include the circuitry for interacting with the antenna(s)to cause transmission of signals via the antenna(s) or to handle receiptof signals received via the antenna(s). These signals may be transmittedor received by the apparatus 200 using any of a number of Internet,Ethernet, cellular, satellite, or wireless technologies, such as IEEE802.11, Code Division Multiple Access (CDMA), Global System for Mobiles(GSM), Universal Mobile Telecommunications System (UMTS), Long-TermEvolution (LTE), Bluetooth® v1.0 through v5.0, Bluetooth Low Energy(BLE), infrared wireless (e.g., IrDA), ultra-wideband (UWB), inductionwireless transmission, Wi-Fi, near field communications (NFC), WorldwideInteroperability for Microwave Access (WiMAX), radio frequency (RF),RFID, or any other suitable technologies.

The API circuitry 210 includes hardware components designed orconfigured to receive, process, generate, and transmit data, such asmachine learning model selections, machine learning model experimentcreation inputs, machine learning model experiment run types, andmachine learning model input data paths. In some embodiments, the APIcircuitry 210 may be embodied as Flux API 124, Flux API Hosts 142, FluxAPI 174, or any other suitable device or circuitry.

In some embodiments, the API circuitry 210 may be in communication witha computing device (e.g., one or more user devices 110, one or moreprogrammatic client devices 112, one or more remote server devices 114,one or more remote data storage devices 116, one or more client devices122, Flux UI 128, one or more user devices 170, one or more clientservices 172, Flux UI 178, any other suitable computing device orcircuitry, or a combination thereof) and thus configured to receive amachine learning model selection from the computing device.

In some embodiments, the API circuitry 210 may be further configured toreceive a machine learning model experiment creation input from thecomputing device or from a different computing device. In someinstances, the machine learning model experiment creation input maycomprise a machine learning model experiment name, a machine learningmodel experiment description, a machine learning model execution engineselection, at least one machine learning model scoring file, and atleast one machine learning model scoring data file.

In some instances, the at least one machine learning model scoring datafile may comprise a trained machine learning model. The trained machinelearning model is configured to be trained, or has been trained (e.g.,by machine learning model training circuitry 222), using a machinelearning model training dataset. The machine learning model trainingdataset may comprise a plurality of observations of a target variableand a feature vector comprising a plurality of associated observationsof non-target variables observed at a time relative to the targetvariable.

In some embodiments, the API circuitry 210 may be further configured toreceive a machine learning model experiment run type from the computingdevice or from a different computing device. In some instances, themachine learning model experiment run type is one of batch scoring,real-time scoring, or training. In some embodiments, the API circuitry210 may be further configured to receive a machine learning model inputdata path from the computing device or from a different computingdevice.

In some embodiments, the API circuitry 210 may receive one or more ofthe machine learning model selection, the machine learning modelexperiment creation input, the machine learning model experiment runtype, and the machine learning model input data path from a firstcomputing device (e.g., a user device 110, a programmatic client device112) and, in response, obtain the remainder of the machine learningmodel selection, the machine learning model experiment creation input,the machine learning model experiment run type, and the machine learningmodel input data path from a second computing device (e.g., a remotedata storage device 116). In some embodiments, the API circuitry 210 mayreceive the machine learning model selection, the machine learning modelexperiment creation input, the machine learning model experiment runtype, and the machine learning model input data path from the memory204.

In some embodiments, the API circuitry 210 may be in communication withand configured to transmit data (e.g., one or more of the machinelearning model selection, the machine learning model experiment creationinput, the machine learning model experiment run type, and the machinelearning model input data path) to one or more computing devices orcircuitries, such as DaaS 126, one or more remote devices 130 (e.g.,Gdoop Spark server 132), Gdoop resource manager 146, one or more FluxUtility Hosts 148 (e.g., scheduler 150, one or more Flux workers 152,job monitor 154), batch execution circuitry 164 (e.g., one or more batchexecution engines 180), artifact store 176, or any other suitable deviceor circuitry.

The machine learning model lifecycle management circuitry 212 includeshardware components designed or configured to receive, process,generate, and transmit data, such as machine learning model experimentsand machine learning model scores. In some embodiments, the machinelearning model lifecycle management circuitry 212 may be embodied asmachine learning model lifecycle management circuitry 162 or any othersuitable device or circuitry.

In some embodiments, machine learning model lifecycle managementcircuitry 212 may be in communication with the API circuitry 210 andthus configured to receive the machine learning model selection, themachine learning model experiment creation input, the machine learningmodel experiment run type, the machine learning model input data path,or a combination thereof from the API circuitry 210. In someembodiments, the machine learning model lifecycle management circuitry212 may be configured to determine a machine learning model executionengine based on the machine learning model execution engine selectionand the machine learning model experiment run type. For example, themachine learning model lifecycle management circuitry 212 may beconfigured to determine a machine learning model execution engine inaccordance with the techniques described above with reference to FIGS.1A-1D.

In some embodiments, the machine learning model lifecycle managementcircuitry 212 may be configured to retrieve input data based on themachine learning model input data path. For example, the machinelearning model lifecycle management circuitry 212 may be configured toretrieve the input data from a computing device (e.g., a data storagedevice or database), memory (e.g., memory 204), or both. In someinstances, the input data may comprise a feature vector. The featurevector may comprise a most recent observation for each of a plurality offeatures. Each of the plurality of features may be identified by afeature identifier and associated with an observation timestamp.

In some embodiments, the machine learning model lifecycle managementcircuitry 212 may be configured to execute a machine learning modelexperiment based on the machine learning model execution engine, themachine learning model scoring file, the at least one machine learningmodel scoring data file, and the input data. For example, the machinelearning model lifecycle management circuitry 212 may be configured toexecute a machine learning model experiment by transmitting instructionsand data to the batch execution circuitry 214 or the real-time executioncircuitry 216.

In some embodiments, the machine learning model lifecycle managementcircuitry 212 may be configured to generate one or more machine learningmodel scores based on the machine learning model experiment. In someinstances, the one or more machine learning model scores may beassociated with the machine learning model experiment and the inputdata. In some embodiments, the one or more machine learning model scoresmay comprise one or more score statistics. In some instances, each scorestatistic of the one or more machine learning model scores may comprisea feature identifier, a score timestamp, a score, and an experimentidentifier.

In some embodiments, the machine learning model lifecycle managementcircuitry 212 may be configured to: retrieve the input data stored in amemory location identified by the machine learning model input datapath; load the at least one machine learning model scoring data fileinto a machine learning model execution environment based on the machinelearning model execution engine selection; and execute the machinelearning model scoring file to create a machine learning model dataframe comprising the one or more machine learning model scores.

In some embodiments, wherein the machine learning model experiment runtype is batch scoring, the machine learning model lifecycle managementcircuitry 212 may be further configured to: determine that the machinelearning model execution engine is a batch execution engine; generate abatch execution control signal indicative of an electronic instructionto execute the machine learning model experiment based on the batchexecution engine; transmit the batch execution control signal to batchexecution circuitry 214, wherein batch execution circuitry 214 isconfigured to execute the machine learning model experiment based on thebatch execution engine; receive batch execution data from the batchexecution circuitry 214, wherein the batch execution data is indicativeof one or more results (e.g., scores, predictions, statistics, or otherdata) of the machine learning model experiment executed by the batchexecution circuitry 214 based on the batch execution engine; andgenerate the one or more machine learning model scores based on thebatch execution data.

In some embodiments, wherein the machine learning model experiment runtype is real-time scoring, the machine learning model lifecyclemanagement circuitry 212 may be further configured to: determine thatthe machine learning model execution engine is a real-time executionengine; generate a real-time execution control signal indicative of anelectronic instruction to execute the machine learning model experimentbased on the real-time execution engine; transmit the real-timeexecution control signal to real-time execution circuitry 216, whereinreal-time execution circuitry 216 is configured to execute the machinelearning model experiment based on the real-time execution engine;receive real-time execution data from the real-time execution circuitry216, wherein the real-time execution data is indicative of one or moreresults of the machine learning model experiment executed by thereal-time execution circuitry 216 based on the real-time executionengine; and generate the one or more machine learning model scores basedon the real-time execution data.

The batch execution circuitry 214 includes hardware components designedor configured to receive, process, generate, and transmit data, such asbatch machine learning model experiments and batch machine learningmodel scores. In some embodiments, the batch execution circuitry 214 maybe embodied as batch execution circuitry 164, as described above withreference to machine learning model lifecycle management circuitry 212,or as any other suitable device or circuitry.

The real-time execution circuitry 216 includes hardware componentsdesigned or configured to receive, process, generate, and transmit data,such as real-time machine learning model experiments and real-timemachine learning model scores. In some embodiments, the real-timeexecution circuitry 216 may be embodied as real-time execution circuitry166, as described above with reference to machine learning modellifecycle management circuitry 212, or as any other suitable device orcircuitry.

The publishing circuitry 218 includes hardware components designed orconfigured to receive, process, generate, and transmit data, such aspublishing selections, access data, and security credentials. In someembodiments, the publishing circuitry 218 may be configured to receive amachine learning model experiment publishing selection. In someinstances, the machine learning model experiment publishing selectionmay comprise one or more access levels respectively associated with oneor more users or sets of users. For example, the machine learning modelexperiment publishing selection may comprise a “read only” access levelfor a first set of users and a “read/write” access level for a secondset of users.

In some embodiments, the publishing circuitry 218 may be furtherconfigured to enable access to the machine learning model experiment andthe one or more machine learning model scores based on the machinelearning model experiment publishing selection. In one illustrativeexample, the publishing circuitry 218 may be configured to permit a setof users to view the machine learning model experiment and the one ormore machine learning model scores. In another illustrative example, thepublishing circuitry 218 may be configured to lock in a first set ofusers (e.g., an engineering team) with the machine learning modelexperiment and its defined schema of the feature vector, while enablinga second set of users (e.g., a data science team) to vary the machinelearning model experiment separately without affecting the productionenvironment.

In some embodiments, the publishing circuitry 218 may be furtherconfigured to publish the machine learning model experiment and the oneor more machine learning model statistics in a feature catalog. In someinstances, the feature catalog may comprise a plurality of featuredefinitions.

The UI circuitry 220 includes hardware components designed or configuredto receive, process, generate, and transmit data, such as UI data andcontent (e.g., Flux UI data and content). In some embodiments, the UIcircuitry 220 may be embodied as Flux UI 128, Flux UI 178, or any othersuitable device or circuitry.

In some embodiments, the UI circuitry 220 may be configured to generateUI data, such as the user interface display screens illustrated in FIGS.7-21. In some embodiments, the UI circuitry 220 may be in communicationwith a display device (e.g., input-output circuitry 206, a user device110, a user device 170, or a display device communicatively coupledthereto) and thus configured to transmit UI data (e.g., Flux UI displayscreen data) to the display device. For example, the UI circuitry 220may be configured to generate UI data and transmit the generated UI datato the input-output circuitry 206, and the input-output circuitry 206may be configured to receive the UI data and display the received UIdata on one or more display screens.

The machine learning model training circuitry 222 includes hardwarecomponents designed or configured to receive, process, generate, andtransmit data, such as trained machine learning models. In someembodiments, such as when the machine learning model experiment run typeis training, the machine learning model training circuitry 222 may beconfigured to train the trained machine learning model using a machinelearning model training dataset comprising a plurality of observationsof a target variable and a feature vector comprising a plurality ofassociated observations of non-target variables observed at a timerelative to the target variable.

In some embodiments, each of the API circuitry 210, machine learningmodel lifecycle management circuitry 212, batch execution circuitry 214,real-time execution circuitry 216, publishing circuitry 218, UIcircuitry 220, and machine learning model training circuitry 222 mayinclude a separate processor, specially configured field programmablegate array (FPGA), application specific interface circuit (ASIC), orcloud utility to perform the above functions. In some embodiments, thehardware components described above with reference to API circuitry 210,machine learning model lifecycle management circuitry 212, batchexecution circuitry 214, real-time execution circuitry 216, publishingcircuitry 218, UI circuitry 220, and machine learning model trainingcircuitry 222, may, for instance, utilize communications circuitry 208or any suitable wired or wireless communications path to communicatewith a user device (e.g., one or more user devices 110, one or more userdevices 170), a programmatic client device (e.g., one or moreprogrammatic client devices 112), a client service (e.g., one or moreclient services 172), a server device (e.g., one or more remote serverdevices 114), a data storage device (e.g., one or more remote datastorage devices 116), each other, or any other suitable circuitry ordevice.

In some embodiments, one or more of the API circuitry 210, machinelearning model lifecycle management circuitry 212, batch executioncircuitry 214, real-time execution circuitry 216, publishing circuitry218, UI circuitry 220, and machine learning model training circuitry 222may be hosted locally by the apparatus 200. In some embodiments, one ormore of the API circuitry 210, machine learning model lifecyclemanagement circuitry 212, batch execution circuitry 214, real-timeexecution circuitry 216, publishing circuitry 218, UI circuitry 220, andmachine learning model training circuitry 222 may be hosted remotely(e.g., by one or more cloud servers) and thus need not physically resideon the apparatus 200. Thus, some or all of the functionality describedherein may be provided by a third party circuitry. For example, theapparatus 200 may access one or more third party circuitries via anysort of networked connection that facilitates transmission of data andelectronic information between the apparatus 200 and the third partycircuitries. In turn, the apparatus 200 may be in remote communicationwith one or more of the API circuitry 210, machine learning modellifecycle management circuitry 212, batch execution circuitry 214,real-time execution circuitry 216, publishing circuitry 218, UIcircuitry 220, and machine learning model training circuitry 222.

As described above and as will be appreciated based on this disclosure,embodiments of the present disclosure may be configured as systems,apparatuses, methods, computer program products, services (e.g., clientservices, machine learning model lifecycle management as-a-service),backend network devices, other suitable devices, and combinationsthereof. Accordingly, embodiments may comprise various means includingentirely of hardware or any combination of software with hardware.Furthermore, embodiments may take the form of a computer program producton at least one non-transitory computer-readable storage medium havingcomputer-readable program instructions (e.g., computer software)embodied in the storage medium. Any suitable computer-readable storagemedium may be utilized including non-transitory hard disks, CD-ROMs,flash memory, optical storage devices, or magnetic storage devices. Aswill be appreciated, any computer program instructions and/or other typeof code described herein may be loaded onto a computer, processor orother programmable apparatus's circuitry to produce a machine, such thatthe computer, processor, or other programmable circuitry that executesthe code on the machine creates the means for implementing variousfunctions, including those described herein.

The one or more user devices 110, user devices 170, programmatic clientdevices 112, client services 172, remote server devices 114, remote datastorage devices 116, client devices 122, remote devices 130, and otherdevices and circuitries described with reference to FIGS. 1A-1D may beembodied by one or more computing devices or systems that also mayinclude processing circuitry, memory, input-output circuitry, andcommunications circuitry. For example, a remote data storage devices 116may be a database server on which computer code (e.g., C, C++, C#, java,a structured query language (SQL), a data query language (DQL), a datadefinition language (DDL), a data control language (DCL), a datamanipulation language (DML)) is running or otherwise being executed byprocessing circuitry. In another example, a user device 110 may be alaptop computer on which an app (e.g., a GUI application provided byFlux UI 128) is running or otherwise being executed by processingcircuitry. In yet another example, a user device 110 may be a smartphoneon which an app (e.g., a webpage browsing app) is running or otherwisebeing executed by processing circuitry. As it relates to operationsdescribed in the present disclosure, the functioning of these devicesmay utilize components similar to the similarly named componentsdescribed above with respect to FIG. 2. Additional description of themechanics of these components is omitted for the sake of brevity. Thesedevice elements, operating together, provide the respective computingsystems with the functionality necessary to facilitate the communicationof data with the machine learning model lifecycle management systemdescribed herein.

Having described specific components of example devices involved in thepresent disclosure, example procedures for managing machine learningmodel lifecycles are described below in connection with FIGS. 3-21.

FIGS. 3A-3B illustrate examples of the machine learning model lifecyclemanagement system's data model. FIG. 3A illustrates an example machinelearning model lifecycle management system architecture data model 300in accordance with some example embodiments. Blocks of data model 300are the Model 301, the Scheduled Run 302, the Experiment 303, the Run304, and the Score 305.

In some embodiments, a machine learning model 301 may represent themachine learning task. The machine learning model 301 may comprise ahuman friendly title (e.g., User Lifetime Value) and one or more targetvariables. In some embodiments, a machine learning model 301 record maycomprise a machine learning model identifier, a title, a permalink, adescription, target variables, and an identifier of a publishedexperiment.

In some embodiments, a machine learning model experiment 303 mayrepresent a method for predicting the target variables which comprisethe machine learning model. The machine learning model experiment 303may represent a certain set of features provided to a certain algorithmwith a certain set of hyper-parameters. In some embodiments, a machinelearning model experiment 303 record may comprise a machine learningmodel experiment identifier, a machine learning model identifier, atitle, a description, and an indication of a machine learning modelengine.

In some embodiments, a machine learning model experiment scheduled run302 may represent a scheduled execution of a machine learning modelexperiment. Machine learning model scheduled runs come in multipleforms, including the training run, batch scoring runs, and always-onstreaming runs. In some embodiments, a machine learning model experimentscheduled run 302 record may comprise a scheduled run identifier, anexperiment identifier, a schedule, an indication of execution engine,and an input mechanism.

In some embodiments, a machine learning model experiment run 304 mayrepresent the execution of a machine learning model experiment. Machinelearning model runs come in multiple forms, including the training run,batch scoring runs, and always-on streaming runs. In some embodiments, amachine learning model experiment run 304 record may comprise a runidentifier, an experiment identifier, an indication of execution engine,and an input mechanism. In some embodiments, the machine learning modelexperiment run 304 record may further comprise a status, metrics, and ascheduled run identifier.

In some embodiments, a machine learning model score 305 may represent aparticular prediction of a target variable produced by the execution ofa machine learning model experiment. In some embodiments, a machinelearning model score 305 record may comprise an entity identifier, anexperiment identifier, a score timestamp, and a score or, in someinstances, a prediction.

FIG. 3B illustrates an example machine learning model lifecyclemanagement system architecture data model 320 in accordance with someexample embodiments. Blocks of data model 320 are the Model 321, theExperiment 323, the Run 324, and the Trained Artifact 326.

In some embodiments, a machine learning model 321 may represent themachine learning task. The machine learning model 321 may comprise atitle, one or more target variables, and one or more identifiers. Insome embodiments, a machine learning model 321 record may comprise amachine learning model identifier, a title, a permalink, a description,target variables, and an identifier of a published experiment.

In some embodiments, a machine learning model experiment 323 mayrepresent a method for predicting the target variables which comprisethe machine learning model. The machine learning model experiment 323may represent a certain set of features provided to a certain algorithmwith a certain set of hyper-parameters. In some embodiments, a machinelearning model experiment 323 record may comprise a description, one ormore model parameters, and an indication of a machine learning modelengine.

In some embodiments, a machine learning model experiment run 324 mayrepresent the execution of a machine learning model experiment. Machinelearning model runs come in multiple forms, including the training run,batch scoring runs, and always-on streaming runs. In some embodiments, amachine learning model experiment run 324 record may comprise a runtype, input data, and a status.

In some embodiments, a machine learning model trained artifact 326 mayrepresent a particular trained artifact produced by the execution of amachine learning model experiment. In some embodiments, a machinelearning model trained artifact 326 record may comprise a location.

In some embodiments, a machine learning model may have many machinelearning model experiments over the course of its life, and may activelyexecute multiple machine learning model experiments simultaneously tocompare performance. However, consumers of the machine learning modelwill not benefit from having a plurality of scores for the same record.Rather, a single machine learning model experiment is designated suchthat its scores are published or made accessible for consumption bydownstream systems. While all scores are stored and made available tocertain clients, a downstream consumer asking for the scores for aparticular machine learning model will be given the scores for thecurrently published machine learning model experiment.

FIG. 4 illustrates an example feature catalog system architecture inaccordance with some example embodiments. In embodiments, a systemarchitecture of a feature catalog system 400 (referred to herein andelsewhere as QED or Quantum Engineered Data) maintains a catalog offeature definitions and curating a data corpus where incoming data hasbeen validated before being made available for use by models. Featurecatalog system 400 includes an API and user interface 401, a workercomponent 402, and data stores 403 (e.g., HDFS data store 403A, HBasedata store 403B). The workers 402 extract data from individual ETLs fromupstream data sources 404 and persist them in data stores 403.Downstream clients 405 make requests through the API 401. The dataset isassembled and delivered via HDFS 406. Downstream clients 405 may explorethe feature catalog through the user interface 401.

In some embodiments, the feature catalog system 400 may decouple thedefinition and generation of a feature from the definition andgeneration of a dataset. By modeling individual features first ratherthan datasets, QED shift the focus to the quality and correctness ofindividual feature calculations while simultaneously promotingreusability. To this end, the features defined in QED do not make anymodel-specific decisions, such as those surrounding imputation. Suchdecisions are “last-mile feature engineering,” and such work belongs inthe model engine.

In some embodiments, a feature defined in QED may comprise the detailsfor how to calculate the feature as well as information about thecadence at which observations of the feature should be taken, and a setof validations to apply to new sets of observations to ensure dataquality before incorporating the new measurements into the generalcorpus. Candidate observation sets that fail any of these validationsare isolated and information about the data abnormality is sent to theappropriate parties for further investigation.

In some embodiments, there may be significant overlap between thefeatures of a machine learning model that, for example, predictsbookings for a hypothetical distribution of device rendered objects fora merchant and a machine learning model that predicts bookings of anactual distribution of device rendered objects that is ready to launch.Similarly, in some embodiments, there may be significant overlap betweenthe dataset used to predict a user's value over the next week and thatused to predict a user's lifetime value. In QED such features may bedesigned and calculated once. When the distinct datasets are requested,they may be built from the common corpus of data.

In some embodiments, QED may define two types of dataset: the trainingdataset and the scoring dataset. In some instances, a machine learningmodel training dataset may be built from a set of observations of atarget variable and corresponding feature vectors observed at some timerelative to the target variable. In some instances, a machine learningmodel scoring dataset may be built from a set of feature vectorscontaining the most recent observation of each feature.

In some embodiments, there may be a temporal relationship between thetarget variable and the feature vector in a training dataset. That is,the values in the feature vector may have been observed some time Xbefore the value of the target variable was observed. This might be astandard offset for all vectors, or it may be dynamic based on when thetarget variable was observed. For instance, if the target variablerepresents bookings for a deal 30 days after launch, then the featurevector may be observed as of the deal launch date, or, in someinstances, 30 days before the deal launch to account for the salescycle. If the target variable represents the current lifetime bookingsof a user, the feature vectors may all be measured 90 days before today.

In some embodiments, to facilitate building these machine learning modeltraining datasets, QED may maintain historical observations of featurevalues. In some embodiments, when generating a machine learning modelscoring dataset, only the most recent observation may be used. However,when building a machine learning model training dataset, each featurevector may use the historical observations of the features to build thefeature vector as it would have looked at some point relative to whenthe target variable was observed. Machine learning model scoringdatasets may be much more straightforward. The requested feature vectormay be returned with current observations.

FIG. 5 illustrates an example data model 500 of an example featurecatalog system in accordance with some example embodiments. Features ofeach entity are grouped into ETLs, which are the basic unit of dataextraction operations. Features within the same ETL are closely relatedand share the same data source and derivation logic (such as SQL queriesor a request to external services). In some embodiments, each ETL maydefine the following metadata:

Schedule: the machine learning model execution run used by thescheduler;

Identifier: a unique identifier field of the entity of the ETL;

Source query: the source query if the data source is a database,otherwise a string stating the source. For time-dependent features, thisis the scoring query for daily runs, and training query is defined in anadditional list;

Schema: the data schema that specifies the datatype of each field, andwhether the field may be null. The schema is versioned so that changesare tracked and verification does not fail when there is a schemachange;

Verification rules: a list of statistical rules to be checked for eachfield; and

Aggregation rules (optional): aggregations to be performed after data isextracted from upstream sources.

The data generated by each ETL may be stored independent in therespective data stores.

In some embodiments, the example data model 500 may depict two entities:Entity A 501; and Entity B 502. Entity A 501 comprises three ETLs, andEntity B 502 comprises two ETLs. Ellipses indicate metadata of the ETLsand dashed outlines indicates optional. Boxes indicate features. It willbe appreciated that all ETLs of the same entity share the same uniqueidentifier. In this example, Entity A 501 has declared that one of itsfeatures is a foreign key to Entity B 502.

Example Operations for Managing Machine Learning Model Lifecycles

FIG. 6 illustrates an example flowchart 600 that contains exampleoperations for managing a machine learning model lifecycle according toan example embodiment. The operations illustrated in FIG. 6 may, forexample, be performed by one or more components described with referenceto machine learning model lifecycle management system 102 shown in FIG.1, by a computing device (e.g., user device 110, programmatic clientdevice 112, remote server device 114, remote data storage device 116,client device 122, user device 170, client services 172) incommunication with machine learning model lifecycle management system102; by apparatus 200 shown in FIG. 2; by any other component describedherein; or by any combination thereof. In some embodiments, the variousoperations described in connection with FIG. 6 may be performed by theapparatus 200 by or through the use of one or more of processingcircuitry 202, memory 204, input-output circuitry 206, communicationscircuitry 208, API circuitry 210, machine learning model lifecyclemanagement circuitry 212, batch execution circuitry 214, real-timeexecution circuitry 216, publishing circuitry 218, UI circuitry 220, andmachine learning model training circuitry 222, any other suitablecircuitry, or any combination thereof.

As shown by block 602, the apparatus 200 includes means, such as APIcircuitry 210 or the like, for receiving a machine learning modelselection. In some embodiments, the apparatus 200 may receive themachine learning model selection from a computing device (e.g., userdevice 110, programmatic client device 112), UI circuitry (e.g., UIcircuitry 220, Flux UI 128, Flux UI 178), or memory (e.g., memory 204)as described in more detail with reference to FIGS. 1-5 and 7-21.

As shown by block 604, the apparatus 200 includes means, such as APIcircuitry 210 or the like, for receiving a machine learning modelexperiment creation input. The machine learning model experimentcreation input may comprise, for example, a machine learning modelexperiment name, a machine learning model experiment description, amachine learning model execution engine selection, at least one machinelearning model scoring file, and at least one machine learning modelscoring data file. In some embodiments, the apparatus 200 may receivethe machine learning model experiment creation input from a computingdevice (e.g., user device 110, programmatic client device 112), UIcircuitry (e.g., UI circuitry 220, Flux UI 128, Flux UI 178), or memory(e.g., memory 204) as described in more detail with reference to FIGS.1-5 and 7-21.

As shown by block 606, the apparatus 200 includes means, such as APIcircuitry 210 or the like, for receiving a machine learning modelexperiment run type. The machine learning model experiment run type maybe, for example, one of batch scoring, real-time scoring, or training.In some embodiments, the apparatus 200 may receive the machine learningmodel experiment run type from a computing device (e.g., user device110, programmatic client device 112), UI circuitry (e.g., UI circuitry220, Flux UI 128, Flux UI 178), or memory (e.g., memory 204) asdescribed in more detail with reference to FIGS. 1-5 and 7-21.

As shown by block 608, the apparatus 200 includes means, such as APIcircuitry 210 or the like, for receiving a machine learning modelexperiment input data path. In some embodiments, the apparatus 200 mayreceive the machine learning model experiment input data path from acomputing device (e.g., user device 110, programmatic client device112), UI circuitry (e.g., UI circuitry 220, Flux UI 128, Flux UI 178),or memory (e.g., memory 204) as described in more detail with referenceto FIGS. 1-5 and 7-21.

As shown by block 610, the apparatus 200 includes means, such as machinelearning model lifecycle management circuitry 212 or the like, fordetermining a machine learning model execution engine based on themachine learning model execution engine selection and the machinelearning model experiment run type. In some embodiments, the apparatus200 may determine the machine learning model execution engine asdescribed in more detail with reference to FIGS. 1-5 and 7-21.

As shown by block 612, the apparatus 200 includes means, such as machinelearning model lifecycle management circuitry 212 or the like, forretrieving input data based on the machine learning model input datapath. In some embodiments, the apparatus 200 may retrieve the input dataas described in more detail with reference to FIGS. 1-5 and 7-21.

As shown by block 614, the apparatus 200 includes means, such as machinelearning model lifecycle management circuitry 212 or the like, forexecuting a machine learning model experiment based on the machinelearning model execution engine, the machine learning model scoringfile, the at least one machine learning model scoring data file, and theinput data. In some embodiments, the apparatus 200 may execute themachine learning model experiment as described in more detail withreference to FIGS. 1-5 and 7-21.

As shown by block 616, the apparatus 200 includes means, such as machinelearning model lifecycle management circuitry 212 or the like, forgenerating, based on the machine learning model experiment, one or moremachine learning model scores associated with the machine learning modelexperiment and the input data. In some embodiments, the apparatus 200may generate the one or more machine learning model scores as describedin more detail with reference to FIGS. 1-5 and 7-21.

In some embodiments, the operations described with reference to FIG. 6(e.g., operations 602, 604, 606, 608, 610, 612, 614, 616) may notnecessarily occur in the order shown in FIG. 6, and in some cases one ormore of the operations shown in FIG. 6 may occur substantiallysimultaneously, or additional steps may be involved before, after, orbetween any of the operations shown in FIG. 6.

As noted above, computing systems, computing apparatuses, computingmethods, and computer program products are described herein that managemachine learning model lifecycles. By doing so, users and clients areable to avoid the traditional problems surrounding machine learningmodel lifecycle management. For instance, through performance of theabove operations, users may save time when exploring machine learningmodels due to the publication of the machine learning model experimentand the one or more machine learning model scores and further due to thesimplified content exploration provided by the user interfaces describedherein. Further, the user interfaces and data structures describedherein simplify the navigation and searching of machine learning modelscores and statistics and, in some instances, are more intuitive forusers. Additionally, in some embodiments, the machine learning modellifecycle management system described herein improves the accuracy ofmachine learning prediction algorithms and the precision ofrecommendations (e.g., per single user) over traditional systems.

FIG. 6 thus illustrates a flowchart describing the operation of variouscomputing systems (e.g., machine learning model lifecycle managementsystem 102 described with reference to FIG. 1), computing apparatuses(e.g., apparatus 200 described with reference to FIG. 2), computingmethods, and computer program products according to example embodimentscontemplated herein. It will be understood that each operation offlowchart 600, and combinations of operations in flowchart 600, may beimplemented by various means, such as hardware, firmware, processor,circuitry, and/or other devices associated with execution of softwareincluding one or more computer-executable program code instructions. Forexample, one or more of the procedures described above may be performedby execution of program code instructions. In this regard, the programcode instructions that, when executed, cause performance of theprocedures described above may be stored by a non-transitorycomputer-readable storage medium (e.g., memory 204) of a computingapparatus (e.g., apparatus 200) and executed by a processor (e.g.,processing circuitry 202) of the computing apparatus. As will beappreciated, any such program code instructions may be loaded onto acomputer or other programmable apparatus (e.g., hardware) to produce amachine, such that the resulting computer or other programmableapparatus implements the functions specified in the operations offlowchart 600. These program code instructions may also be stored in acomputer-readable memory that may direct a computer or otherprogrammable apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture, the execution of which implements the functionsspecified in the operations of flowchart 600. The program codeinstructions may also be loaded onto a computer or other programmableapparatus to cause a series of operations to be performed on thecomputer or other programmable apparatus to produce acomputer-implemented process such that the instructions executed on thecomputer or other programmable apparatus provide operations forimplementing the functions specified in the operations of flowchart 600.

The flowchart operations described with reference to FIG. 6 supportcombinations of means for performing the specified functions andcombinations of operations for performing the specified functions. Itwill be understood that one or more operations of the flowchart, andcombinations of operations in the flowchart, may be implemented byspecial purpose hardware-based computer systems which perform thespecified functions, or combinations of special purpose hardware andcomputer instructions.

Use Case

Having described example embodiments in general terms, the followingexample embodiments are provided to further illustrate a use case ofsome example embodiments. In some instances, the following exampleembodiments provide examples of how the machine learning model lifecyclemanagement system disclosed herein may manage the lifecycles of machinelearning models. In some instances, the following example embodimentsprovide examples of the ways in which the machine learning modellifecycle management system disclosed herein may generate user interfacedata based on the various data or electronic information disclosedherein.

“Flux” Use Case

“Flux” is an illustrative example use case wherein the machine learningmodel lifecycle management system disclosed herein may generate aninteractive user interface application referred to herein as “Flux” anddescribed in greater detail below with reference to FIGS. 7-21.

Recent years have seen a boom in machine learning applications in manyaspects of the information technology industry. To facilitate theseapplications, automated, scalable machine learning infrastructures arebeing developed at many companies. As machine learning becomes thedriving force of the daily operation of companies within the informationtechnology sector, infrastructure that enables automated, scalablemachine learning is a core component of the systems of many largecompanies. Various systems and products are being built, offered, andopen sourced. Numerous aspects of an e-commerce company's business maybe driven by machine learning.

One great challenge of these machine learning systems is the integrationof offline model prototypes into online production environments.Oftentimes, the tools used by data scientists who develop modelprototypes are different from and sometimes incompatible with theproduction systems. A typical way to address this problem is to undergoa manual process that re-implements models built with research tools,with libraries that are compatible with production systems. However,this presents challenges for both data scientists and engineering teamssince it is time-consuming and prone to errors.

Another problem is that the contract between data scientists andengineers constantly evolves over time. On the one hand, in thedevelopment stage of machine learning algorithms, fast prototyping andparameter tuning needs short feedback loops. A data scientist may decideto add or drop certain features, or build the model with anotheralgorithm, or change the format of the target variable on a veryfrequent cadence. On the other hand, the engineering team that isresponsible for generating input for or consuming the output of themodel want the model to be stable, and might need to update theirservices to accommodate changes to the model when they happen. To avoidproduction issues, careful planning and coordination may be necessary. Agood machine learning infrastructure may need to have a mechanism thatversions the contract, enabling fast iteration while facilitating themaintenance of a stable production environment.

Furthermore, depending on the use case, an offline batch execution, or areal-time or streaming paradigm might be desired. A machine learningsystem may need to support both paradigms. A machine learning systemalso needs to be easily scalable, since the load may vary drastically incertain scenarios. A good machine learning platform should also beextensible. Machine learning is evolving at a rapid pace, and librariesand frameworks can appear and become outdated quickly. Depending ontheir background, data scientists might prefer different libraries,languages, or both. A well-designed platform should be able to onboardnew machine learning libraries with ease. In addition, a machinelearning infrastructure should provide functionalities such asmonitoring and archiving. These functionalities will enable keyactivities such as AB testing of models and offline analytics.

Traditional systems present too much hindrance in integrating with acompany's internal systems and thus may not be suitable for thecompany's purpose. In addition, traditional systems may be tied to aspecific library and thus the range of supported algorithms may belimited. For example, some traditional frameworks are built forparticular libraries, such as Spark MLlib and H2O Steam, and can bedifficult to extend. Both Spark MLlib and H2O Steam rely on specificalgorithm implementations and thus it can be costly and difficult toonboard other libraries.

In contrast to traditional systems, Flux is a computing system thatsolves these problems of machine learning model productionization byproviding a set of tools and services for managing the lifecycle ofthese models, including execution of training and scoring. Flux providesREST APIs for programmatic access to models and their executions. Fluxsupports multiple model engines and offers great flexibility for modelprototyping. Model execution may be done in batch or real-time modes. Inaddition, Flux integrates machines learning models into the productionsystem and provides monitoring and archiving for the models. Further,Flux solves the scalability issue and provides a seamless collaborationbetween data scientists and engineers. Flux expedites the deployment,execution, and monitoring of machine learning models. Flux focuses onenabling data scientists to build model prototypes with languages andtools they are most proficient in, and integrating the models into theenterprise production system. Flux manages the lifecycle of deployedmodels, and executes them in distributed batch mode, or exposes them asmicroservices for real-time use cases. Flux's design focuses onautomation and improved management, scalability, and extensibility. Fluxis the central system for supervised machine learning tasks and has beensupporting multiple teams. For example, Flux has productionized twentymodels of seven teams and onboarded additional clients. These modelshave been utilized in many important areas of company operations, suchas demand prediction and lead scoring. Typical algorithms includegeneralized linear models and decision tree ensembles such as randomforest and gradient boosting decision trees. In some embodiments, Fluxmay not provide feature engineering as a separate stage in the pipelineat a system level, but rather Flux may leave feature engineering toindividual model implementations.

In some embodiments, to ease the interaction with the machine learningmodel lifecycle management system for users of user devices, the machinelearning model lifecycle management system may comprise a “Flux” userinterface (e.g., Flux UI 128, Flux UI 178). The user interface mayprovide functionalities that allow clients to create Models,Experiments, and Runs, and publish and delete Experiments. The userinterface may also present information about each Run, where key events(e.g., start, end, success, and failure) are logged. In one illustrativeexample, the user interface may be used by data scientist clients whoneed more user-friendly interfaces.

FIGS. 7-21 illustrate various example user interface display screens700, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800,1900, 2000, and 2100 in accordance with some example embodiments. It isalso noted that all or some of the information presented by the exampleuser interface display screens discussed herein may be based on datathat is received, generated and/or maintained by one or more componentsof apparatus 200, such as UI circuitry 220. In some embodiments, one ormore external systems (e.g., a remote cloud computing and/or datastorage system) may also be leveraged to provide at least some of thefunctionality discussed herein. For example, FIG. 21 shows a screenshotof the Flux UI showing a Model, an Experiment, and a list of batchscoring Runs.

In some embodiments, data science teams may use Flux to quickly iterateand improve on models that have already been productionized. Systemsthat consume machine learning model outputs may integrate with Fluxwithout the need to delve into details of how certain model works. Bothdata science teams and engineering teams have enjoyed multiple benefitsfrom the Flux platform. Using Flux, machine learning modelproductionization that used to take weeks, if not months, now only takesa few days to happen. In one illustrative example, the batch executionengine has enabled high-throughput model prediction that wasunattainable in single-box environments. For example, Flux has reducedthe prediction time of one model for 14 million feature vectors from 14hours down to 1.5 hours, which is critical for the model to bepractically useful in production. In another illustrative example, thereal-time execution engine has made fast prediction possible for modelsthat requires low latency. Depending on the specific algorithm andimplementation, the response time may vary, but the real-time executionengine adds only a small overhead to the model prediction, typicallytens of to a few hundred milliseconds.

There are many advantages of these and other embodiments describedherein. For example, Flux provides cleaner seams between data scientistsand engineers. In addition, Flux infrastructure is not tied to aspecific library and does not limit the range of algorithms that itsupports. For example, Flux's extensible architecture currently supportsmodels that are defined in Cognitive Foundry (a Java machine learninglibrary), or models that are implemented in R (with access to the vastnumber of libraries available in R). Accordingly, Flux provides a largesubset of features and presents reduced hindrance in integrating with acompany's internal systems. Further, Flux provides generic, extensiblesupport for machine learning and is not tied to a specific library.Accordingly, the range of algorithms supported by Flux is not limited.Further still, Flux is designed to be easily extensible. To onboard newmachine learning algorithms and libraries, only a small set ofinterfaces, and in some instances an adapator, need to be implemented.

CONCLUSION

While various embodiments in accordance with the principles disclosedherein have been shown and described above, modifications thereof may bemade by one skilled in the art without departing from the teachings ofthe disclosure. The embodiments described herein are representative onlyand are not intended to be limiting. Many variations, combinations, andmodifications are possible and are within the scope of the disclosure.Alternative embodiments that result from combining, integrating, and/oromitting features of the embodiment(s) are also within the scope of thedisclosure. Accordingly, the scope of protection is not limited by thedescription set out above, but is defined by the claims which follow,that scope including all equivalents of the subject matter of theclaims. Each and every claim is incorporated as further disclosure intothe specification and the claims are embodiment(s) of the presentdisclosure. Furthermore, any advantages and features described above mayrelate to specific embodiments, but shall not limit the application ofsuch issued claims to processes and structures accomplishing any or allof the above advantages or having any or all of the above features.

In addition, the section headings used herein are provided forconsistency with the suggestions under 37 C.F.R. § 1.77 or to otherwiseprovide organizational cues. These headings shall not limit orcharacterize the disclosure set out in any claims that may issue fromthis disclosure. For instance, a description of a technology in the“Background” is not to be construed as an admission that certaintechnology is prior art to any disclosure in this disclosure. Neither isthe “Summary” to be considered as a limiting characterization of thedisclosure set forth in issued claims. Furthermore, any reference inthis disclosure to “disclosure” or “embodiment” in the singular shouldnot be used to argue that there is only a single point of novelty inthis disclosure. Multiple embodiments of the present disclosure may beset forth according to the limitations of the multiple claims issuingfrom this disclosure, and such claims accordingly define the disclosure,and their equivalents, that are protected thereby. In all instances, thescope of the claims shall be considered on their own merits in light ofthis disclosure, but should not be constrained by the headings set forthherein.

Also, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other devices or components shown or discussed as coupled to, or incommunication with, each other may be indirectly coupled through someintermediate device or component, whether electrically, mechanically, orotherwise. Other examples of changes, substitutions, and alterations areascertainable by one skilled in the art and could be made withoutdeparting from the scope disclosed herein.

Many modifications and other embodiments of the disclosure set forthherein will come to mind to one skilled in the art to which theseembodiments pertain having the benefit of teachings presented in theforegoing descriptions and the associated figures. Although the figuresonly show certain components of the apparatus and systems describedherein, it is understood that various other components may be used inconjunction with the machine learning model lifecycle management system.Therefore, it is to be understood that the disclosure is not to belimited to the specific embodiments disclosed and that modifications andother embodiments are intended to be included within the scope of theappended claims. For example, the various elements or components may becombined, rearranged, or integrated in another system or certainfeatures may be omitted or not implemented. Moreover, the steps in anymethod described above may not necessarily occur in the order depictedin the accompanying drawings, and in some cases one or more of the stepsdepicted may occur substantially simultaneously, or additional steps maybe involved. Although specific terms are employed herein, they are usedin a generic and descriptive sense only and not for purposes oflimitation.

1.-20. (canceled)
 21. An apparatus comprising a processor and anon-transitory memory storing program instructions, wherein thenon-transitory memory and the program instructions are configured to,with the processor, cause the apparatus to: receive a first userselection of a machine learning model experiment creation button on amachine learning model user interface rendered on a client device; causerendering, on the client device, a machine learning model experimentcreation user interface comprising a machine learning model experimentname input field, a machine learning model experiment description inputfield, a machine learning model execution engine selection field, and amachine learning model scoring file upload field; receive, via themachine learning model experiment creation user interface, machinelearning model experiment creation input comprising a machine learningmodel experiment name, a machine learning model experiment description,a machine learning model execution engine selection, and a machinelearning model scoring file; and cause rendering, on the client device,a machine learning model experiment user interface based at least inpart on the machine learning model experiment creation input.
 22. Theapparatus of claim 21, wherein the machine learning model experimentuser interface comprises a machine learning model experiment runcreation button, wherein the non-transitory memory and the programinstructions are configured to, with the processor, cause the apparatusto: in response to receiving a second user selection of the machinelearning model experiment run creation button, cause rendering, on theclient device, a machine learning model experiment run creation userinterface comprising a machine learning model experiment run selectionfield and a machine learning model input data path input field.
 23. Theapparatus of claim 22, wherein the non-transitory memory and the programinstructions are configured to, with the processor, cause the apparatusto: receive, via the machine learning model experiment run creation userinterface, a machine learning model experiment run type and a machinelearning model input data path, wherein the machine learning modelexperiment run type is one of batch scoring or training.
 24. Theapparatus of claim 23, wherein the non-transitory memory and the programinstructions are configured to, with the processor, cause the apparatusto: determine a machine learning model execution engine based on themachine learning model execution engine selection and the machinelearning model experiment run type; and retrieve input data based on themachine learning model input data path.
 25. The apparatus of claim 24,wherein the non-transitory memory and the program instructions areconfigured to, with the processor, cause the apparatus to: execute amachine learning model experiment based on the machine learning modelexecution engine, the machine learning model scoring file, and the inputdata.
 26. The apparatus of claim 25, wherein the non-transitory memoryand the program instructions are configured to, with the processor,cause the apparatus to: generate one or more machine learning modelscores based on the machine learning model experiment, wherein the oneor more machine learning model scores are associated with the machinelearning model experiment and the input data.
 27. The apparatus of claim26, wherein the one or more machine learning model scores comprise oneor more score statistics, and wherein each score statistic of the one ormore machine learning model scores comprises a feature identifier, ascore timestamp, a score, and an experiment identifier.
 28. Acomputer-implemented method comprising: receiving a first user selectionof a machine learning model experiment creation button on a machinelearning model user interface rendered on a client device; causingrendering, on the client device, a machine learning model experimentcreation user interface comprising a machine learning model experimentname input field, a machine learning model experiment description inputfield, a machine learning model execution engine selection field, and amachine learning model scoring file upload field; receiving, via themachine learning model experiment creation user interface, machinelearning model experiment creation input comprising a machine learningmodel experiment name, a machine learning model experiment description,a machine learning model execution engine selection, and a machinelearning model scoring file; and causing rendering, on the clientdevice, a machine learning model experiment user interface based atleast in part on the machine learning model experiment creation input.29. The computer-implemented method of claim 28, wherein the machinelearning model experiment user interface comprises a machine learningmodel experiment run creation button, wherein the computer-implementedmethod further comprises: in response to receiving a second userselection of the machine learning model experiment run creation button,causing rendering, on the client device, a machine learning modelexperiment run creation user interface comprising a machine learningmodel experiment run selection field and a machine learning model inputdata path input field.
 30. The computer-implemented method of claim 29,further comprises: receiving, via the machine learning model experimentrun creation user interface, a machine learning model experiment runtype and a machine learning model input data path, wherein the machinelearning model experiment run type is one of batch scoring or training.31. The computer-implemented method of claim 30, further comprising:determining a machine learning model execution engine based on themachine learning model execution engine selection and the machinelearning model experiment run type; and retrieving input data based onthe machine learning model input data path.
 32. The computer-implementedmethod of claim 31, further comprising: executing a machine learningmodel experiment based on the machine learning model execution engine,the machine learning model scoring file, and the input data.
 33. Thecomputer-implemented method of claim 32, further comprising: generatingone or more machine learning model scores based on the machine learningmodel experiment, wherein the one or more machine learning model scoresare associated with the machine learning model experiment and the inputdata.
 34. The computer-implemented method of claim 33, wherein the oneor more machine learning model scores comprise one or more scorestatistics, and wherein each score statistic of the one or more machinelearning model scores comprises a feature identifier, a score timestamp,a score, and an experiment identifier.
 35. A computer program productcomprising at least one non-transitory computer-readable storage mediumhaving computer-readable program code portions stored therein, thecomputer-readable program code portions comprising an executable portionconfigured to: receive a first user selection of a machine learningmodel experiment creation button on a machine learning model userinterface rendered on a client device; cause rendering, on the clientdevice, a machine learning model experiment creation user interfacecomprising a machine learning model experiment name input field, amachine learning model experiment description input field, a machinelearning model execution engine selection field, and a machine learningmodel scoring file upload field; receive, via the machine learning modelexperiment creation user interface, machine learning model experimentcreation input comprising a machine learning model experiment name, amachine learning model experiment description, a machine learning modelexecution engine selection, and a machine learning model scoring file;and cause rendering, on the client device, a machine learning modelexperiment user interface based at least in part on the machine learningmodel experiment creation input.
 36. The computer program product ofclaim 35, wherein the machine learning model experiment user interfacecomprises a machine learning model experiment run creation button,wherein the computer-readable program code portions comprise theexecutable portion configured to: in response to receiving a second userselection of the machine learning model experiment run creation button,cause rendering, on the client device, a machine learning modelexperiment run creation user interface comprising a machine learningmodel experiment run selection field and a machine learning model inputdata path input field.
 37. The computer program product of claim 36,wherein the computer-readable program code portions comprise theexecutable portion configured to: receive, via the machine learningmodel experiment run creation user interface, a machine learning modelexperiment run type and a machine learning model input data path,wherein the machine learning model experiment run type is one of batchscoring or training.
 38. The computer program product of claim 37,wherein the computer-readable program code portions comprise theexecutable portion configured to: determine a machine learning modelexecution engine based on the machine learning model execution engineselection and the machine learning model experiment run type; andretrieve input data based on the machine learning model input data path.39. The computer program product of claim 38, wherein thecomputer-readable program code portions comprise the executable portionconfigured to: execute a machine learning model experiment based on themachine learning model execution engine, the machine learning modelscoring file, and the input data.
 40. The computer program product ofclaim 39, wherein the computer-readable program code portions comprisethe executable portion configured to: generate one or more machinelearning model scores based on the machine learning model experiment,wherein the one or more machine learning model scores are associatedwith the machine learning model experiment and the input data.